First, robots.txt is simply a plain-text file that a Web publisher or web master should put in the root directory of their website. (Example: http://www.yahoo.com/robots.txt. It’s Yahoo Robots.txt file, take a look.) The text files includes instructions that tell indexing spiders, or ‘robots’, what content and directories on that website they may, or may not, look at. In simple one line: this file allows you to hide files or directories that you don’t wish the search engine spiders to find.

Here’s an example of a robots.txt file:

User-agent: *
Disallow:

Above code inform every search engines that you can crawl all web site’s page and content.

Here is another example how to block a directory or a particular page through Robots.txt

User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html

Posted by  seo expert services : SEobyExpert.com a leading Internet Marketing Company offering affordable SEO Packages to grow your business.