All about Robots.txt  
 
robots.txt generator

List the directories you want to manage be sure to put a / at the beginning and end of each one. Example: /cgi-bin/

    

 
Robots.txt examples:

To allow all robots/spiders index your website
User-agent: *
Disallow:

Download this sample here to use on your site

_________________________________

To exclude all robots/spiders from your website
User-agent: *
Disallow: /

To exclude all robots/spiders from part of your website
User-agent: *
Disallow: /cgi-bin/

To exclude a single robot/spider
User-agent: Googlebot
Disallow: /

 


 
 
  A more advanced free robots.txt generator can be found at sitesubmit.ca

 

 

 

     
     

Copyright © 2006 XPC Technology Services. Vancouver, Canada  All Rights Reserved Terms of Service

All logos and trademarks in this site are property of their respective owner.