What's new

How to create Robots.txt file ?

ebenzunlimited

Moderator
A robots.txt file will help search engines properly index your page. It tells the search engine where not to go - the opposite of a sitemap. This is the first thing search engines look for when they index your site and is an essential component o

f a well-organized and professionally created website



Steps To follow:
Open Notepad
Write the first line. All robots.txt files start with this line:
User-Agent: *
Decide what areas you do not want to have indexed.

* If you want all parts of your site to be indexed, add this line:
Disallow:
* Add lines for every area that you do want to be indexed. For example, if you do not want images and a private folder indexed:
Disallow: /images/
Disallow: /private/

Save the file. Use the name "robots.txt"

Upload to the main directory of your website.

Note:
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it
source:wikihow.com

For More details visit: http://www.robotstxt.org/
 
robot.png
A robots.txt file will help search engines properly index your page. It tells the search engine where not to go - the opposite of a sitemap. This is the first thing search engines look for when they index your site and is an essential component o

f a well-organized and professionally created website



Steps To follow:
Open Notepad
Write the first line. All robots.txt files start with this line:
User-Agent: *
Decide what areas you do not want to have indexed.

* If you want all parts of your site to be indexed, add this line:
Disallow:
* Add lines for every area that you do want to be indexed. For example, if you do not want images and a private folder indexed:
Disallow: /images/
Disallow: /private/

Save the file. Use the name "robots.txt"

Upload to the main directory of your website.

Note:
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it
source:wikihow.com

For More details visit: http://www.robotstxt.org/
 
Top