You can learn a lot about a search engine by scraping its results. It’s the only easy way you can get an hourly or daily record of exactly what Google, Bing or Yahoo! (you know, back when Yahoo! was a search engine company) show their users. It’s also the easiest way to track your keyword rankings.

Like it or not, whether you use a third-party tool or your own, if you practice SEO then you’re scraping search results.

If you follow a few simple rules, it’s a lot easier than you think.

The problem with scraping

Automated scraping — grabbing search results using your own ‘bot’— violates every search engine’s terms of service. Search engines sniff out and block major scrapers.

If you ever perform a series of searches that match the behavior of a SERP crawler, Google and Bing will interrupt your search with a captcha page. You have to enter the captcha or perform whatever test the page requires before performing another query.

That (supposedly) blocks bots and other scripts from automatically scraping lots of pages at once.

The reason? Resources. A single automated SERP scraper can perform tens, hundreds or even thousands of queries per second. The only limitations are bandwidth and processing power. Google doesn’t want to waste server cycles on a bunch of sweaty-palmed search geeks’ Python scripts. So, they block almost anything that looks like an automatic query.

Your job, if you ever did anything like this, which you wouldn’t, is to buy or create software that does not look like an automatic query. Here are a few tricks my friend told me:

Stay on the right side of the equation

Note that I said “almost” anything. The search engines aren’t naive. Google knows every SEO scrapes their results. So does Bing. Both engines have to decide when to block a scraper. Testing shows that equation to make the block/don’t block decision balances:

• Potential server load created by the scraper.
• The potential load created by blocking the scraper.
• The query ‘space’ for the search phrase.

At a minimum, any SERP bot must have the potential of tying up server resources. If it doesn’t, the search engine won’t waste the CPU cycles. It’s not worth the effort required to block the bot.

So, if you’re scraping the SERPs, you need to stay on the right side of that equation: Be so unobtrusive that, even if you’re detected, you’re not worth squashing.

Disclaimer

Understand, now, that everything I talked about in this article is totally hypothetical. I certainly don’t scrape Google. That would violate their terms of service.

And I’m sure companies like AuthorityLabs and SERPBuddy have worked out special agreements for hourly scraping of the major search engines. But I have this… friend… who’s been experimenting a bit, testing what’s allowed and what’s not, see…

Cough.

How I did my test

I tested all these theories with three Python scripts. All of them:

1. Perform a Google search.
2. Download the first page of results.
3. Then downloads the next 4 pages.
4. Saves the pages for parsing.

Script #1 had no shame. It hit Google as fast as possible and didn’t attempt to behave like a ‘normal’ web browser.

Script #2 was a little embarrassed. It pretended to be Mozilla Firefox and only queried Google once every 30 seconds.

Script #3 was downright bashful. It selected a random user agent from a list of 10, and paused between queries for anywhere between 15 and 60 seconds.

The results

Script #3 did the best. That’s hardly a surprise. But the difference is:

• Script #1 was blocked within 3 searches.
• Script #2 was blocked within 10 searches.
• Script #3 was never blocked, and performed 150 searches. That means it pulled 5 pages of ranking data for 150 different keywords.

There’s no way any of these scripts fooled Google. The search engine had to know that scripts 1, 2 and 3 were all scrapers. But it only blocked 1 and 2.

My theory: Script 3 created so small a burden that it wasn’t worth it for Google to block it. Just as important, though, was the fact that script 3 didn’t make itself obvious. Detectable? Absolutely. I didn’t rotate IP addresses or do any other serious concealment. But script 3 behaved like it was, well, embarrassed. And a little contrition goes a long way. If you acknowledge you’re scraping on the square and behave yourself, Google may cut you some slack.

The rules

Based on all of this, here are my guidelines for scraping results:

1. Scrape slowly. Don’t pound the crap out of Google or Bing. Make your script pause for at least 20 seconds between queries.
2. Scrape randomly. Randomize the amount of time between queries.
3. Be a browser. Have a list of typical user agents (browsers). Choose one of those randomly for each query.

Follow all three of these and you’re a well-behaved scraper. Even if Google and Bing figure out what you’re up to, they leave you alone: You’re not a burglar. You’re scouring the gutter for loose change. And they’re OK with that.

More articles by this author

Dear Google: This is war

Dear Google: With your announcement yesterday, you've become the enemy. My...

Read More >>

A Scalable Content Strategy in a Post-Panda World

The Panda has raged across the globe now, leaving a...

Read More >>