A robots.txt file will help search engines properly index your page. It tells the search engine where not to go - the opposite of a sitemap. This is the first thing search engines look for when they index your site and is an essential component o
f a well-organized and professionally created website
Steps To follow:
Write the first line. All robots.txt files start with this line:
Decide what areas you do not want to have indexed.
* If you want all parts of your site to be indexed, add this line:
* Add lines for every area that you do want to be indexed. For example, if you do not want images and a private folder indexed:
Save the file. Use the name "robots.txt"
Upload to the main directory of your website.
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it
For More details visit: http://www.robotstxt.org/