# General Reference: http://www.robotstxt.org/robotstxt.html # Yahoo understands wildcards at the file level. It also has a habit of # repeatedly reindexing removed files, causing tons of 404 errors. # Reference: http://www.webmasterworld.com/robots_txt/3144662.htm User-Agent: yahoo! slurp Allow: * Disallow: /Events/*.txt User-Agent: * Disallow: /NeverThisOne.html Disallow: /CommentForm-Tracked2006.php