Author Topic: How to create Robots.txt file ?  (Read 3450 times)

0 Members and 1 Guest are viewing this topic.

Offline ebenzunlimited

  • Lead Admin
  • *****
How to create Robots.txt file ?
« on: May 24, 2012, 10:28:25 PM »
A robots.txt file will help search engines properly index your page. It tells the search engine where not to go - the opposite of a sitemap. This is the first thing search engines look for when they index your site and is an essential component o

f a well-organized and professionally created website



Steps To follow:
Open Notepad
Write the first line. All robots.txt files start with this line:
User-Agent: *
Decide what areas you do not want to have indexed.

* If you want all parts of your site to be indexed, add this line:
Disallow:
* Add lines for every area that you do want to be indexed. For example, if you do not want images and a private folder indexed:
Disallow: /images/
Disallow: /private/

Save the file. Use the name "robots.txt"

Upload to the main directory of your website.

Note:
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it
source:wikihow.com

For More details visit: http://www.robotstxt.org/
Insane I Lived♥ Sane I Died♥<img src='http://i253.photobucket.com/albums/hh74/reallytired2/family%20pictures/buckeye%20stuff/clap.gif' />

Offline ebenzunlimited

  • Lead Admin
  • *****
How to create Robots.txt file ?
« Reply #1 on: June 06, 2012, 02:35:32 AM »
A robots.txt file will help search engines properly index your page. It tells the search engine where not to go - the opposite of a sitemap. This is the first thing search engines look for when they index your site and is an essential component o

f a well-organized and professionally created website



Steps To follow:
Open Notepad
Write the first line. All robots.txt files start with this line:
User-Agent: *
Decide what areas you do not want to have indexed.

* If you want all parts of your site to be indexed, add this line:
Disallow:
* Add lines for every area that you do want to be indexed. For example, if you do not want images and a private folder indexed:
Disallow: /images/
Disallow: /private/

Save the file. Use the name "robots.txt"

Upload to the main directory of your website.

Note:
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it
source:wikihow.com

For More details visit: http://www.robotstxt.org/
Insane I Lived♥ Sane I Died♥<img src='http://i253.photobucket.com/albums/hh74/reallytired2/family%20pictures/buckeye%20stuff/clap.gif' />

 

* Post Updates

WPX is giving out 6 months of free WordPress hosting this week by obasimiracle
[November 25, 2020, 01:56:47 AM]


Re: Outlook PST Merge by ruth less
[November 11, 2020, 07:22:38 AM]


Re: How to Convert OLM to PST by priyankaarya
[November 10, 2020, 01:03:10 PM]

Inside: 3P Techies Blog

* Newest Techies

Get Updates


Sign up to get latest updates delivered to your inbox. No Spam, We Promise!

Get Hosting!

a Faster web hosting service

Copyright 3rd Planet Techies. All rights Reserved.

Top || Mobile ||Advertise || Contact Us || Privacy