TL;DR: Google’s anti-scraping measures are making keyword tracking tools less precise, and Google Search Console can’t completely fill the gap. Keep using the tools but cross-check them and verify key rankings manually.
Here’s bit of history for you: In 1997, the manager of the band Jefferson Starship reportedly wanted the group’s official website to be listed higher than page four in a search engine. As the story goes, that’s when the term “search engine optimization” (SEO) was born.
Back then, the world wide web was a tiny fraction of its current size, and keyword ranking checks were done manually. That didn’t last long, though. As the web grew and efficiency was necessary, companies were created for keyword tracking, and they built Google scraping tools. An entire industry grew up around optimizing for search engines, like Google, Yahoo, and Ask Jeeves. Scraping Google search engine results pages (SERPs) for ranking position, ranking URL, and other intel has been a practice for a long time despite Google saying in its Terms of Service that the practice goes against its policy and can lead to termination of your access or termination of your account.
Source: Google Terms of Service
Companies, such as Semrush, Ahrefs, Moz, and Advanced Web Ranking (AWR), offer keyword tracking tools to show SEOs exactly where their client’s websites are ranking for specific keywords without the time and labor investment of manually checking the SERPs every day. While those tools have not been perfect, they’ve been widely accepted, used, and paid for because they are incredibly helpful for tracking organic ranking progress over time.
Google’s Anti-Scraping Measures
In recent years, Google has been aggressively deploying anti-scraping measures to confound the tools that SEOs rely on. We’ve seen random Captcha check boxes while manually checking the SERPs for keyword positioning or to see which competitors are winning and how. This summer, Google began routing the links in its search results through google.com/goto passthrough URLs. Google confirmed the rollout and said it takes steps against ‘evolving forms of abuse,’ and industry observers see it as another obstacle for scraping tools.
According to a blog post by Advanced Web Ranking titled, “Why Your Tracked Rankings No Longer Match What You See in Google,” the search giant has also started changing the search results it shows to suspected web crawling bots versus humans. If the engine suspects the query is from a bot, it shows results for a much broader geography than it would for a human, which can change the SERP significantly.
For example, if a human searches for a service from a Raleigh IP address, Google’s natural inclination to show localized results will typically limit the websites to businesses in the Raleigh, Durham, and Cary area. But if Google suspects a bot is querying the same keyword, it may show results for all of North Carolina or possibly an even wider geography.
In a conversation with a help desk employee at AWR, we were told to change our tracking from national to a specific city for better results. Unfortunately, working with national and international clients means that focusing on the search results for a single city is going to provide skewed results. In that email thread, we were assured the company is developing a new work around.
In this game of whack-a-mole between the keyword tracking tools and Google, the search giant is currently winning.
Why Not Just Use Google Search Console for Keyword Tracking?
Unlike the other keyword tracking tools, we can’t force Google Search Console (GSC) to track a specific keyword over time. While GSC will display keywords the site is ranking for, we have no control over which keywords it shows. It may show the keyword you’re interested in, or only variations of it, since GSC reports the exact phrases searchers typed. It may also not show the keyword at all, even if your site ranks for it, because GSC caps how many queries it displays and hides low-volume queries for privacy reasons.
During times of high SERP volatility, a site’s position for a keyword can swing sharply from one day to the next. GSC only shows what it surfaces on the day you look, so if a keyword is missing or has dropped, you can’t tell whether your rankings changed or the report simply didn’t display it. Keyword tracking tools that check the same specific keyword every day give you a consistent record, so you can see the fluctuation as it happens.
So, Does This Mean Keyword Tracking Tools Are Useless?
No, but for now, we need to view the data they give us with their current limitations in mind. Be willing to manually check the rankings, cross-reference the different tools, and put in a little extra work to find the most accurate intel you can. Contact the SEO/GEO team here at ROI Revolution to learn how we can help.
Sources
- Search Engine Journal, 20+ Years of SEO: A Brief History of Search Engine Optimization
- Search Engine Roundtable, Google May Be More Successful In Blocking Scrapers & Tracking Tools
- Search Engine Roundtable, Confirmed: Google Search Rolling Out google.com/goto Tracking Parameters
- Advanced Web Ranking, Why Your Tracked Rankings No Longer Match What You See in Google
