What’s the difference between the robots.txt file and a noindex meta tag? Both of these tags are designed to tell the search engine crawlers to “not index” a URL on a website. But are they really the same? In many situations, adding a disallow directive in the robots.txt file will have the same effect as using a meta noindex tag on a web page. But, there are times when you should NOT use the robots.txt file and use the meta noindex tag, instead. Let’s look at the differences.
Posts published by “Bill Hartzer”
SEO Consultant and CEO of Hartzer Consulting, based in Texas. Hartzer Consulting provides technical SEO audits for websites.