|
Prevent Robots from Indexing Parts of Your SiteAs you know search engines such as Google, Yahoo, and MSN will send robots (bots, for short) to spider and index web pages that it finds in the internet. There may be parts of your website that you don't want the search engines to find. Bots do use up some of your webhost bandwidth allowances as they do hit pages on your site. So if you might save some bandwidth by disallow them. The most common scenario is to disallow a bots to index certain subdirectories. To do this add a robots.txt file at the root of your website with contents such as ... User-agent: * Don't put blank lines in between. This record in the robots.txt file will disallow all user-agents (or bots) from indexing the directories listed. If there is a web page that you don't want bots to index, you can put this meta tag in the <head> section of the HTML page... <meta name="robots" content="noindex,nofollow"> This tells robots to not index nor follow the links on this particular web page. Whether the bots obey these directives or not is up to the bots. But most well-known bots do in fact obey them. Googlebot for sure does obey them as Google has indicated so in this page. Here are more information about robots.txt file and the robots meta tag.
|
 |
 |
| Roadmap | Lessons | Tutorials | Videos | Books | Links | About |
|
|
