Last week, Google started a great under-the-hood series on how their search engine works.

Part 1 gives some good information on how you can fine tune what parts of your site Google will index and how they will appear in a user’s search results.

The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels — the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.

—
Link