Follow Along

RSS Feed Join Us on Twitter On Facebook

Get Engaged


Featured Article

Under the hood; the SEOGadget Keyword Research, BetaUnder the hood; the SEOGadget Keyword Research,  BetaA Tag-team Review from Level 343 In the SEO world, new tools are a dime a dozen. Some make the grade,...

Latest Comments

Related Reading

Our Sponsors


Join Us

Technical SEO: The Frog that Tromped Google Webmaster Tools
Written by Terry Van Horne
Tuesday, 26 July 2011 13:12

(co-written by SEO Gal)

You’re absolutely confident you’re doing things right. You’ve gone down your personal SEO checklist: yep, yep, yep. Everything is running perfectly… Of course, that was back when you first set up your online shop. You know, back before you got busy with clients and had nothing but time to spend on your site.

Here it is, a year or two later, and you’re still thinking you’re site is clean. Every time you’ve changed a URL, you’ve done the nifty 301 redirect. Maybe you’re running Wordpress, and have made sure you don’t have that annoying duplicate content issue because of comment pagination. You’ve even kept your eye on Google Webmaster Tools, which says your site is good – and everybody knows how great GWT is at reporting issues, right (cough)?

Well, the next time you have a spare minute, scoot on over to Screaming Frog and download the SEO Spider. It takes about five minutes all told to download, set up and run. The free version alone, which crawls 500 URIs at a time, will probably be enough to send you scrambling for time to clean up the mess, or at least a friendly industry contact to help you take care of it quick.

Screaming Frog SEO Spider Review

Not Your Average Crawler

The Screaming Frog SEO Spider is one powerful little tool, completely replacing GWT. Not only does it completely blow GWT’s error reporting out of the water, it does a whole lot more for technical SEO:

  • Get real response codes (client and server side)
  • Easily check 3XX redirects
  • Check for URI issues
  • View site architecture based on page level of crawled URIs
  • Etc, etc, etc and more!


Thought you dealt with WP duplicate content issues?

Do you have your canonicals in place? Awesome. However, if you have your “Reply To” button turned on, Google is likely indexing duplicate content. On one scan of a WP site, I found tons of duplicate content issues with a [?replytocom] parameter. The number of times each post was duplicated depends on, you guessed it, the number of comments to the post.

What a nightmare. This means if you post an awesome article and get 30 comments, you could, perceivably, end up with 30 [?replytocom] duplicates. How do you fix it? If you don’t know, find out here.

Is your content as clean as you think it is?

Is your blogging getting repetitive? SEO Spider pulls the title, description, H1 and H2 tags for your quick perusal. So what, you say?

Think about it. Often, we think we’re being original with titles, topics, and snappy lingo. However, if you’ve been writing about your topic as long as I’ve been writing about mine, I can almost guarantee you have at least one article covering the same issue as another.

Now, if you’ve done your content development correctly, your title should reflect the content on the page. Being able to view these little bits and pieces in one place lets you scan them. What you’ll probably find (I did, on one site), are content gaps. Content gaps are like thinning hair… if it’s missing, it’s noticeable. This is an excellent chance to find out where you could fill out your hair content.

Are you staying true to your key term focus?

It’s easy, especially when you’re busy with clients and churning out content, to lose track of your original focus. Screaming Frog SEO Spider gives you a list of per page key terms based on the meta keyword tag.

Now, most people completely ignore this tag, and that’s up to you. However, if you use them as a way to remind yourself what a page is about, they can come in pretty handy. As well, some CMS use tags to fill in this particular area. If you do use this meta tag, it can help pinpoint areas where you might have lost topical focus (although, not necessarily lost relevance).

Where SEO Spider Really Shines

Filters. If you’re crawling 10,000 pages looking for specific issues, it’d be nice to be able to remove pages from your view that don’t have those issues, right? I mean, what if you know you have pages that are missing titles and you want to go fill them in real quick like?

With SEO Spider, you can set the filter to only show pages with missing titles, duplicate descriptions, multiple H1, missing alt text, etc. You can filter based on meta data such as noindex or nofollow. Filter by response codes – or, conversely, pages that get no response at all.


Now, technical SEO may not be your “thing”. You may leave the technical audits to someone else. However, whether you’re into it or just into your site, the SEO Spider can really help make sure your online business presence stays on track.

And as an added bonus, here's a quick vid we made to give you a little more insight... Have any questions? Do feel free to leave them in the comments;

Terry Van Horne -

Terry is an old school SEO/IM geek that works out of International Website Builders and the founder of - You can also hook up with him on Twitter. It seems he's become a regular here on the Trail... so I wanted to, once more, give a great big THANKs to T for hanging out and sharing his experiences.

You can also hook up via


More articles by this author

The Art of Google Link BuildingThe Art of Google Link Building
The aftermath of the Over Optimization Penalty which is Googleze...
Is Google Really De-indexing Free Directories?Is Google Really De-indexing Free Directories?
Today Barry Schwartz reported on a WebmasterWorld thread in which...
Last Updated on Tuesday, 26 July 2011 13:21


0 #1 mattmikulla 2011-07-26 14:21
I've been using Screaming Frog for a few months now. It is awesome!

It provides way more data than xenu and there's a mac version.
0 #2 Jun 2011-07-27 00:58
I love this tool as it's making website audit faster and easier. And I'm just using the free one at that!
0 #3 Lasse Heindorff 2011-07-27 13:19
Thanks, Terry. I also love how Screaming Frog SEO Spider makes finding keywords a sport. Have a competitor with hundreds or even thousands of keyword rich landing pages? Then snatch those keywords quickly with this little tool.
0 #4 Jey Pandian 2011-09-01 13:29
This is a good tool, I enjoy using it. Do you use the paid or the lite version?
0 #5 Robin Jennings 2012-09-28 05:55
I've been using the tool for about 6 weeks and its picked up information that neither GWT nor SEOmoz found- I'm I'm still using the free version.
0 #6 Paul McElwee 2013-03-06 06:50
Terry, This tool seems quite useful and I am happy to have a well composed idea about this tool. I havn't used this tool yet but looking forward to start with the free version. I will definitely share my experiences of using this tool with you.
0 #7 دستگاه حضور و غیاب 2014-04-14 11:46
Thank you very much., You are a good writer.
0 #8 درب اتوماتیک 2014-04-21 11:59
Keep blogging.ur site is good.and finelly ur site going to in high position
0 #9 درب ضد سرقت 2014-04-23 11:34
This is a really good read for me. Must agree that you are one of the coolest blogger I ever saw.
0 #10 تور ترکیه 2014-04-26 11:38
I congratulate you
0 #11 پارتیشن اداری 2014-04-27 11:33
In addition to art that will give you good
I congratulate you
0 #12 کرکره برقی 2014-05-05 09:52
Thanks Richard, it's very understandable that under the circumstances continuing didn't make sense. I'll have a look at your Excel extension.
0 #13 راهبند اتوماتیک 2014-06-08 08:55
Very nice site! cheap goods
0 #14 تور چین 2014-06-11 07:51
This site is very great and indispensable to my thank you to expect continued success with the authorities there are millions of people come easy memlunkalan but I guess I did not bring the language
0 #15 تور ترکیه 2014-06-28 05:38
It was a very useful article.
0 #16 تور استانبول 2014-06-28 05:40
Writing is an art that everyone does it. Congratulate you for having this art
0 #17 تور آنتالیا 2014-06-28 05:41
This site is very great and indispensable to my thank you.
0 #18 دانلود آهنگ 2014-06-28 05:44
i really enjoyed to visit this is a very nice site and i book mark ur site.
0 #19 دانلود 2014-06-28 14:13
ناول دانلود
دانلود نرم افزار
نرم افزار
دانلود بازي
مالتي مديا
پخش کننده
آنتي ويروس
ديوار آتش
مديريت دانلود
اف تي پي
فشرده سازي
بهينه سازي
نرم افزار آموزشي
تم ويندوز
بي کلام
نرم افزار اندرويد
نرم افزار جاوا
نرم افزار جديد
دانلود بازي اندرويد
دانلود فيلم
دانلود سريال
بازي کنسول
سايت ناول دانلود
بازي ماشين
بازي فکري
دانلود جديد
0 #20 دانلود نرم افزار 2014-06-28 14:13
Very nice site! cheap goods

Add comment

Security code

Getting Around the Site

Home - all the latest on SNC
SEO - our collection of SEO articles
Technical SEO - for the geeks
Latest News - latest news in search
Analytics - measure up and convert
RSS Rack - feeds from around the industry
Search - looking for something specific?
Authors - Author Login
SEO Training - Our sister site
Contact Us - get in touch with SNC

What's New?