Back to the homepage Submit your website

Robots.txt

   
The robots.txt file is placed in your www or public_html directory.

So the correct location where to put this file is in www.yourdomainname.com/robots.txt. This file indicates how "deep" the spiders can crawl into your website.

It's possible to keep a spider out of a directory that's called "customers" or a directory like "images".  Obviously it's a request, not a guarantee. And if you don't want the file /secret1960 to be found by anyone, maybe it's better not to put it on the internet anyway, and especially not in your robots.txt file. If you want to protect your information you use a password security like .htaccess or a login-script.

This robot.txt file could look like this :

User-agent: *
Disallow: /setup/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /photos/
Disallow: /dealers/


Sometimes spiders ask you to add the following line:

Allow: spider-xyz

Example

 

Download our example txt file and add it in the root of your website. In the root we mean the www file or also called the http_docs or public_html. It's important that the file is in the correct location and that is :

www.yourwebsitename.com/robots.txt

Click on the file with the right mouse button to download the example text file. (select save or save as). Edit the file and add it to your own website.

robots.txt



Meta tags Summary - Important SEO things to remember


Control the search engine spiders and guide them trough your website.
  1. Read why meta tags are important and how they affect the search engines.
  2. Use our meta tags generator to create the correct meta tags.
  3. Use our website analyzer to get professional advice from the meta tags experts.
  4. Submit your website to the search engines and choose one of our packages:
Metatags Free Trial Order our Local Package Order our Euro Package Order our World Package
Free Trial Local Package Euro Package World Package
       
meta tags. design, tips Tags: meta tags, design, tips
© 2014 - Part of the The MetaTags Company BV - Last update: 25-10-2014