Robots txt

A text file kept in the root of a website. It tells robots, spiders and crawlers where they are allowed to go and not allowed to go on a website.

See Also:

SEO, Page Rank, Deep Linking, Linkage



How Robots.txt works:

Suppose a robot wants to vist a Web site URL(e.g, before doing this it firsts checks for It tries to find User-agent: * Disallow: /

Some Important features of Robots.txt:

Important Syntaxes of Robots.txt:


Copyright © 2012 . All Right Reserved.