What is a robots.txt file and do I need it on my site?

     Need to limit sections of your site from the search engines? No problem. Keep them from crawling your sensitive content, or limit exposure of duplicate content.     

     You may have noticed when reviewing your website usage statistics a reference to a file named “robots.txt”. This is a basic configuration file that robots (search engine spiders) use to note which files and directories they should not index.

Although not necessary, it is important to have the file, robots.txt, in the root of your website (http://www.yourdomain.com/robots.txt). The basic use of the robots.txt is to note which files and directories the robots should not index.  The absence of the file will show 404 errors in your website logs, making it more difficult to parse the stats logs for useful information.

There are 2 components to the syntax of the robots.txt file structure: user agent line and the disallow line.

The syntax is as follows: <FIELD> ":" <VALUE>

User-agent

A specific robot may be referenced in the User-agent syntax.

User-agent: googlebot

A wildcard may also be used to note all robots.

User-agent: *

Disallow

If you have a page named “addresses.html”, and you do not want the robot to index it, add the following line after the User-agent syntax:

Disallow: addresses.html

You may also limit the robot at the directory level. If the directory “/billing” in your website should be restricted from indexing, place the following text after the User-agent syntax.

Disallow: /billing/

A wildcard may also be used to note all files on site.

Disallow: /

How do I make comments in my robots.txt file?

# specifies googlebot as the User-agent.

User-agent: googlebot

It is good practice to have your comments on a separate line than the directive. Also, limit white space at the beginning of directive lines.

Putting robots.txt to use:

If you want to allow ALL robots to visit ALL the pages on your website, you should use the “*” wildcard in your syntax in the robots.txt file.

User-agent: *
Disallow:

Disallow all indexing of your site, enter the following text in your robots.txt file.

User-agent: *
Disallow: /

Disallow robots from indexing pages within a directory named “billing”.

User-agent: *
Disallow: /billing/

Disallow robots from indexing pages within directories named “billing” and “address”.

User-agent: *
Disallow: /billing/
Disallow: /address/

Disallow googlebot from your site.

User-agent: googlebot
Disallow: /

Disallow googlebot from accessing you contact page (contact-us.html).

User-agent: googlebot
Disallow: /contact-us.html

Disallow googlebot from accessing you contact page (about-us/contact-us.html).

User-agent: googlebot
Disallow: /about-us/contact-us.html

Robots.txt in a nutshell:

About the author:

Nathan Drach is the founder of Server Union, LLC. ServerUnion is a US based technology company providing web hosting solutions from big to small. For more information and articles visit: www.serverunion.com