Details on Proposed Search Controls
November 28, 2007
Under the formal rules for robots.txt, a Web site may refuse indexing of the entire site, specific directories or specific pages. However, some search engines have interpreted...

News Web Sites Seek More Search Control
November 28, 2007
Currently, Google Inc., Yahoo Inc. and other top search companies voluntarily respect a Web site's wishes as declared in a text file known as robots.txt, which a search engine...

Webmasters May Shape Search Results
November 28, 2007
A programmer can use a robots.txt file to police the crawlers trying to access a site. It's not that the search engines are better, but it's the people out there making policy...

News websites propose more control over use by search engines
November 29, 2007
Currently, Google Inc., Yahoo Inc. and other top search companies voluntarily respect a website's wishes as declared in a text file known as robots.txt, which a search engine...

BREITBART. COM - Court to Hear Google- Newspaper Fight
November 24, 2006
Powell said the Internet search company strenuously tries to give newspapers the opportunity to be excluded from the site if they want, by telling the robots.txt information...

BREITBART. COM - Court to Hear Google- Newspaper Fight
November 23, 2006
Powell said the Internet search company strenuously tries to give newspapers the opportunity to be excluded from the site if they want, by telling the robots.txt information...

Facebook's Publicly Available Data: A Big Strategic Risk
August 18, 2010 | Read/WriteWeb

Quite possibly the best Robots. txt file ever
July 28, 2010 | digg

Last. fm's robots. txt
July 27, 2010 | Boing Boing

You'll still need permission if you want to crawl Facebook's public content
June 30, 2010 | VentureBeat

Rupert Murdoch vows to take all of Newscorp's websites out of Google, abolish fair...
November 08, 2009 | Boing Boing

Another credit- card number Googled on Blippy: Philip Kaplan explains
April 24, 2010 | VentureBeat

Google Ransom Note
November 08, 2009 | Fimoculous.com

Robotstxt.org
www.robotstxt.org

Robots exclusion standard - Wikipedia, the free encyclopedia
en.wikipedia.org

Robots.txt Information
www.robotstxt.ca

Robots.txt Generator - McAnerin International Inc.
www.mcanerin.com

Robots.txt and Search Indexing - Search Tools Report
www.searchtools.com