Robots txt

A text file kept in the root of a website. It tells robots, spiders and crawlers where they are allowed to go and not allowed to go on a website.

See Also:

SEO, Page Rank, Deep Linking, Linkage

 

 

How Robots.txt works:

Suppose a robot wants to vist a Web site URL(e.g http://www.example.com/welcome.html.), before doing this it firsts checks for http://www.example.com/robots.txt. It tries to find User-agent: * Disallow: /

Some Important features of Robots.txt:

Important Syntaxes of Robots.txt:

 

Copyright © 2012 www.thinkbizsolutions.com . All Right Reserved.