Follow Along

RSS Feed Join Us on Twitter On Facebook

Get Engaged


Featured Article

Under the hood; the SEOGadget Keyword Research, BetaUnder the hood; the SEOGadget Keyword Research,  BetaA Tag-team Review from Level 343 In the SEO world, new tools are a dime a dozen. Some make the grade,...

Latest Comments

Related Reading

Our Sponsors


Join Us

Technical SEO: The Frog that Tromped Google Webmaster Tools
Written by Terry Van Horne
Tuesday, 26 July 2011 05:12

(co-written by SEO Gal)

You’re absolutely confident you’re doing things right. You’ve gone down your personal SEO checklist: yep, yep, yep. Everything is running perfectly… Of course, that was back when you first set up your online shop. You know, back before you got busy with clients and had nothing but time to spend on your site.

Here it is, a year or two later, and you’re still thinking you’re site is clean. Every time you’ve changed a URL, you’ve done the nifty 301 redirect. Maybe you’re running Wordpress, and have made sure you don’t have that annoying duplicate content issue because of comment pagination. You’ve even kept your eye on Google Webmaster Tools, which says your site is good – and everybody knows how great GWT is at reporting issues, right (cough)?

Well, the next time you have a spare minute, scoot on over to Screaming Frog and download the SEO Spider. It takes about five minutes all told to download, set up and run. The free version alone, which crawls 500 URIs at a time, will probably be enough to send you scrambling for time to clean up the mess, or at least a friendly industry contact to help you take care of it quick.

Screaming Frog SEO Spider Review

Not Your Average Crawler

The Screaming Frog SEO Spider is one powerful little tool, completely replacing GWT. Not only does it completely blow GWT’s error reporting out of the water, it does a whole lot more for technical SEO:

  • Get real response codes (client and server side)
  • Easily check 3XX redirects
  • Check for URI issues
  • View site architecture based on page level of crawled URIs
  • Etc, etc, etc and more!


Thought you dealt with WP duplicate content issues?

Do you have your canonicals in place? Awesome. However, if you have your “Reply To” button turned on, Google is likely indexing duplicate content. On one scan of a WP site, I found tons of duplicate content issues with a [?replytocom] parameter. The number of times each post was duplicated depends on, you guessed it, the number of comments to the post.

What a nightmare. This means if you post an awesome article and get 30 comments, you could, perceivably, end up with 30 [?replytocom] duplicates. How do you fix it? If you don’t know, find out here.

Is your content as clean as you think it is?

Is your blogging getting repetitive? SEO Spider pulls the title, description, H1 and H2 tags for your quick perusal. So what, you say?

Think about it. Often, we think we’re being original with titles, topics, and snappy lingo. However, if you’ve been writing about your topic as long as I’ve been writing about mine, I can almost guarantee you have at least one article covering the same issue as another.

Now, if you’ve done your content development correctly, your title should reflect the content on the page. Being able to view these little bits and pieces in one place lets you scan them. What you’ll probably find (I did, on one site), are content gaps. Content gaps are like thinning hair… if it’s missing, it’s noticeable. This is an excellent chance to find out where you could fill out your hair content.

Are you staying true to your key term focus?

It’s easy, especially when you’re busy with clients and churning out content, to lose track of your original focus. Screaming Frog SEO Spider gives you a list of per page key terms based on the meta keyword tag.

Now, most people completely ignore this tag, and that’s up to you. However, if you use them as a way to remind yourself what a page is about, they can come in pretty handy. As well, some CMS use tags to fill in this particular area. If you do use this meta tag, it can help pinpoint areas where you might have lost topical focus (although, not necessarily lost relevance).

Where SEO Spider Really Shines

Filters. If you’re crawling 10,000 pages looking for specific issues, it’d be nice to be able to remove pages from your view that don’t have those issues, right? I mean, what if you know you have pages that are missing titles and you want to go fill them in real quick like?

With SEO Spider, you can set the filter to only show pages with missing titles, duplicate descriptions, multiple H1, missing alt text, etc. You can filter based on meta data such as noindex or nofollow. Filter by response codes – or, conversely, pages that get no response at all.


Now, technical SEO may not be your “thing”. You may leave the technical audits to someone else. However, whether you’re into it or just into your site, the SEO Spider can really help make sure your online business presence stays on track.

And as an added bonus, here's a quick vid we made to give you a little more insight... Have any questions? Do feel free to leave them in the comments;

Terry Van Horne -

Terry is an old school SEO/IM geek that works out of International Website Builders and the founder of - You can also hook up with him on Twitter. It seems he's become a regular here on the Trail... so I wanted to, once more, give a great big THANKs to T for hanging out and sharing his experiences.

You can also hook up via


More articles by this author

The Art of Google Link BuildingThe Art of Google Link Building
The aftermath of the Over Optimization Penalty which is Googleze...
Is Google Really De-indexing Free Directories?Is Google Really De-indexing Free Directories?
Today Barry Schwartz reported on a WebmasterWorld thread in which...
Last Updated on Tuesday, 26 July 2011 05:21


0 #1 mattmikulla 2011-07-26 06:21
I've been using Screaming Frog for a few months now. It is awesome!

It provides way more data than xenu and there's a mac version.
0 #2 Jun 2011-07-26 16:58
I love this tool as it's making website audit faster and easier. And I'm just using the free one at that!
0 #3 Lasse Heindorff 2011-07-27 05:19
Thanks, Terry. I also love how Screaming Frog SEO Spider makes finding keywords a sport. Have a competitor with hundreds or even thousands of keyword rich landing pages? Then snatch those keywords quickly with this little tool.
0 #4 Jey Pandian 2011-09-01 05:29
This is a good tool, I enjoy using it. Do you use the paid or the lite version?
0 #5 Robin Jennings 2012-09-27 21:55
I've been using the tool for about 6 weeks and its picked up information that neither GWT nor SEOmoz found- I'm I'm still using the free version.
0 #6 Paul McElwee 2013-03-05 22:50
Terry, This tool seems quite useful and I am happy to have a well composed idea about this tool. I havn't used this tool yet but looking forward to start with the free version. I will definitely share my experiences of using this tool with you.

Add comment

Security code

Getting Around the Site

Home - all the latest on SNC
SEO - our collection of SEO articles
Technical SEO - for the geeks
Latest News - latest news in search
Analytics - measure up and convert
RSS Rack - feeds from around the industry
Search - looking for something specific?
Authors - Author Login
SEO Training - Our sister site
Contact Us - get in touch with SNC

What's New?

All content and images copyright Search News Central 2014
SNC is a Verve Developments production, the Forensic SEO Specialists- where Gypsies roam.