Meta Robots Tag 101: Blocking Spiders, Cached Pages & More

... them to do so. Now if you DO NOT want your pages in a search engine, then it's time to perhaps break out the meta robots tag, if for some reason the robots.txt alternative isn't suitable. Want to keep a particular page out? Then put this on that page: <meta name="robots" content="noindex ...
Related Searches: No Follow | Noodp | Noydir

Mar. 5, 2007 at 8:48pm Eastern by Danny Sullivan

New SEO Tools: Robots.txt Builder, Term Targeting & Crawl Test

Three new SEO tools came out this week that are worth a look. Dave Naylor released a robots.txt builder tool, while SEOmoz released term targeting and crawl test tools. Some more about the tools, below: The Robots.txt Builder Tool David Naylor, the creator of the tool, explains that one of his clients ...
Related Searches: Robots | Seo Tools | Tools

Apr. 20, 2007 at 12:30pm Eastern by Barry Schwartz

Irony: If Google Can't Reach Your Robots.txt File, It Might Not List Your Site

I reported at the Search Engine Roundtable this morning that Google said if your robots.txt is unreachable, your site might not make it into the Google index. By unreachable, Google means that if your server simply times out and does not return any server response when Googlebot attempts to access ...

Jun. 18, 2008 at 8:34am Eastern by Barry Schwartz

Up Close & Personal With Robots.txt

The Robots.txt Summit at Search Engine Strategies New York 2007 was the latest in a series of special sessions with the intent to open a dialog between search engines representatives and web site publishers. Past summits featured discussion on comment spam on blogs, indexing issues and redirects. The ...
Related Searches: Robots | Sitemap | Xml Sitemap

Apr. 16, 2007 at 1:15pm Eastern by Christine Churchill

Search Illustrated: Blocking Search Engines With Robots.txt

... only gets one copy (the "canonical" version, in search engine parlance). Today's Search Illustrated illustrates how you can use the "robots.txt" file as a "keep out" notice for search engine cawlers: Graphic by Elliance, an eMarketing firm specializing in results-driven ...
Related Searches: Robots.txt File | Robots | Canonical

May. 23, 2007 at 3:25am Eastern by Elliance

Google Offers Robots.txt Generator

Google's rolled out a new tool at Google Webmaster Central, a robots.txt generator. It's designed to allow site owners to easily create a robots.txt file, one of the two main ways (along with the meta robots tag) to prevent search engines from indexing content. Robots.txt generators aren't new. You ...

Mar. 27, 2008 at 5:39pm Eastern by Danny Sullivan

Robots.txt Study Shows Webmasters Favor Google; BotSeer Robots.txt Search Engine Released

... search engines in terms of allowing access to their web sites. An associated BotSeer search engine that allows searching across a collection of robots.txt files was also released. The study looked at which robots or crawlers were listed in a web site's robots.txt file, and Google was listed more often ...

Nov. 15, 2007 at 1:19pm Eastern by Barry Schwartz

Google Enhances Webmaster Central's Robots.txt Analysis Tool

The Google Webmaster Central Blog announced improvements they have made to the robots.txt analysis tool. The tool now recognizes all sitemap declarations and relative URLs. So now the tool will report the validity of all sitemaps URLs plus show data for relative URLs. In addition, Google has expanded ...

Aug. 16, 2007 at 8:58am Eastern by Barry Schwartz

Twelve SEO Mistakes Most Bloggers Make

... ). Peruse your indexed pages in Google using the site: query operator and look for which pages don't deserve to be there. Then disallow them in your robots.txt file. Having multiple homes for your blog. Does your blog have what search engine geeks refer to as "canonicalization" issues? If you ...
Related Searches: Wordpress | External Links | Tag Clouds | Title Tags

Aug. 23, 2007 at 8:27am Eastern by Stephan Spencer

Yahoo!, Google, Microsoft Clarify Robots.txt Support

Today, Google, Yahoo!, and Microsoft have come together to post details of how each of them support robots.txt and the robots meta tag. While their posts use terms like "collaboration" and "working together," they haven't joined together to implement a new standard (as they did ...
Related Searches: Acap

Jun. 3, 2008 at 12:11pm Eastern by Vanessa Fox