Understanding data from the search engine results pages (SERPs) is important for any business owner or SEO professional. Do you wonder how your website performs in the SERPs? Are you curious to know where you rank in comparison to your competitors? Keeping track of SERP data manually can be a time-consuming process. Let’s take a look at a proxy network that can help you can gather information about your website's performance within seconds.

Transcript:

Hey, what's up. Welcome to Hack My Growth. In today's video, we're taking a look at a new web scraper that can be extremely helpful when we are analyzing search results. We recently started exploring Bright Data, a proxy network, as well as web scrapers that allow us to get some pretty cool information that will help when it comes to planning a search marketing or SEO strategy. 

The first thing we need to do is look at the search results.

When we are looking for a term, one of the things that we need to do is always look at the search results. In this case, let's say I wanted to target the term, “SERP Analysis.” The first thing we need to do is look at the search results themselves. Now, I like to use a lot of widgets to help me better understand the search results, but I also always need to scroll down to understand to get a better idea of what's going on. Who's ranking? What type of content's ranking? Why are they ranking? What are the different elements on the page? What are the related search queries that we find on this page? This can take some time and it is an important step, so we need to be going to the search results to understand it.

But what's also helpful is that we're able to extract information and then process it on our own to get even more insights. And we start to layer it with other data, which will help, at least in my case, to build a strategy that's going to work. 

Using Bright Data To Interpret Search Results

Sometimes pulling the data off the search results can be pretty difficult, and that's where APIs can come in. What's cool is you don't have to be a programmer. I don't consider myself a programmer. I consider myself decent at copy and pasting. 

With the help of Bright Data, we can pull a lot of cool data from the search results and layer it with some other data. That gives us deeper insights into the search results so we can plan a better strategy. Sign up for an account if you want to follow along with this video. If not, you can sign up afterward and try it out. It's a reasonable service in order to get the data you want. They have a free trial too, so go ahead and try that.

Dashboard

When you're in your dashboard and you want to start to play around with this, you can go over to the map icon here and click on that, and then we can see I've got a SERP API here. Now, you’ll see configuration settings and this is what's going to allow me to connect a number of things, but I can also check out the SERP API Playground. Now you're going to need this access in order to play around in here because you're going to have your own hostname.

SERP API Playground

If you want to do the playground first to see what you can build, click SERP API Playground under the tab Access Parameters. Now I'm going to take that same query that entered into Google Search, “SERP Analysis,” and I'm going to drop it in here. There are a number of different settings. You can choose the search results that you want to look at - Google, Bing, Yandex, Duck Duck Go. For me, we target mostly Google because it's the big boy. And then we can choose whether we're doing search, maps, trends, or reviews. There are a lot of different settings that you can tick on and off here.

Now for me, I work in Python, I always switch this over to Python, and then what's cool is I can also see the HTML and the JSON output with this query. When I hit search and it's going to do its cool stuff. It's giving me the actual Python that I need to use to call this. But then I get to see the results below, and I can click on each of these elements and it'll pull it up on the side for me. I can see different elements. I can see the “People Also Ask” sections, and I can see the answers there. I can see the search results organically. I can see that there are some videos in this search results, and I can see the related searches. In one quick call, I got a ton of data. 

Now some of you may be thinking, well, I don't know what to do with this JSON-LD data, it's not super helpful to me. And it can be a little bit confusing, but the reality is, it's pretty easy to break down. But to make it even easier, I created a CoLab notebook where you can just plug and play there once you are set up with Bright Data.

What I wanted to do was better understand this query myself. We put the query in Google CoLab. We set up the API call, and then you’ll see the Google search data. Now, this is all nested in JSON, and seeing it like this isn't the most helpful for us, but if I want to make some sense out of this and go use it to plan strategy, I can break it down.

SERP Data Overview in Google CoLab

The first thing I did was build something called SERP Data Overview. This just tells me within a simple table, the search engine I used, the query, how many results are for this term, and a little bit of other information like the language and the page title. What I'm interested in are the organic results. You’ll see the links, (these are all the top 10, in order), the title, the description, and any extensions those pages might have. There is other information - inline text or a star rating, a site link, and it’s starting to show me some of the SERP features.

And then I get to see the rank. You’ll the rank organically and the global rank. Global rank is how it ranks within all the elements on the page. This link ranks number one in organic results, but it ranks sixth on the page. Why is that the case? Well, if we go back to the search results, you’ll notice the Featured Snippet, and the four “People Also Ask” questions. 

The first result is six down on the page, which is very telling for us. This is a very competitive search result, and all of the search results are pushed down based on the different elements that are on the page. 

Again, this is helpful for me to realize, "Man, if I rank number one for that term, I'm number six.” How much energy and focus am I going to have on this term if I'm not going to be able to get it based on competitiveness? I'm starting to get insights now.

What else is cool is that I can see some of the other things that could target, because “People Also Ask” are ranking above position one. I might want to rank or target these terms. What are these questions here? Then I can build specific pieces of content in order to rank in those positions. You see the questions, answers, and links that we can use to do some research.

We also can pull out the related terms. We can say, "Well maybe this term is a little bit too competitive, but here are some related terms I might be able to go after," which is cool too. Now I'm getting some keyword research. I'm understanding the search results a little bit better and I'm getting some ideas of what strategies I could take.

Trend Analysis

Now I also want to use their trend analysis. At the top of the page, you'll see that one of the options is trends. And I can put in the same query, “SERP Analysis,” and I can do the search. Now the thing with this one is you're not going to see HTML and the Python script. You're just going to get this raw JSON data. And a lot of this is nested because there are a lot of different pieces of data here. And again, processing it might be a little bit difficult. This is where CoLab, again can be helpful. 

We're running the query and pulling all the data, and you might be thinking, "This doesn't make any sense." And then we start to process the data and we can get a timeline. Over the last 12 months, we saw that “SERP Analysis” came down and then it popped back up. It's starting to be searched frequently. It's trending upward since August 28th.

We can also pull related keywords from the trend analysis. I can pull keywords and I can see how valuable they are. These are trending up - “SERP Analysis Tools,” “SERP Analysis,” “Free Google SERP Analysis.” All of those are a little bit more important than something like “GPT-3 tutorial.” Again, I start to get some more keyword ideas and some low-hanging fruit that may not be as popular as the target term, but it could be easier for me to rank for.

Now, looking at the related keywords in a table is nice, but we can also visualize it with Python. Python is helpful because we can see which ones stand out, and we know which ones are more important and which ones are less important. 

Related Topics

We can also pull the related topics using Bright Data SERP API, and to me, this might be my favorite part when pairing it with what I know about the search results now.

Again, we can dump this data into a table. I can see the topic title, and I can also see the type. And then if I scroll down, I made some different graphs and we can see some queries around specific topics, some queries related to websites, and some queries related to types of software. We're starting to understand more of the entities involved here, which we know are important. We've talked about structured data and other things on this channel. I can see how they're all grouped together. The larger they are, the more popular they are, the smaller they are, the less popular they are. I can start to get an idea as a whole of what's going on with the search result.

Creating and HTML Report

This is a lot of data to process, and we may want to share it with somebody. We can create an HTML report, using all this information with the help of Python as well. We can run the code and see the analysis of the SERP query. We’ll see what the search results look like, and all the cool data. As you scroll down, you see the related terms, the trend analysis of the query that we're targeting, the related terms, and the related topics. Now I can take this information and look at strategies to rank on this page and understand what's going on within the search results. 

This is a super cool API. It does cost a little bit, but it's super reasonable. I think it's $3 per thousand calls, but then you can go in and start to understand the search results because they're changing so much. Search is changing, and the search results are changing. This is going to help you get a better idea of the content that's ranking, answer some of the questions about why the content is ranking, and show some of the things that you need to do in order to get better results yourself.

If you check out Bright Data SERP API, let me know and send me an email. I'll send you a copy of this CoLab file and then you can plug in your data and begin to have some fun with the search results as well.

If you got any questions, contact us! I'd love to continue this conversation with you. And until next time!