# http://www.paonet.org/robots.txt # Wed Nov 16 15:45:40 PST 2005 # Roger C. Pao <20051116-robots-txt-paonet-org.z.rpao@xoxy.net> # References: # http://www.robotstxt.org/wc/robots.html # http://www.kollar.com/robots.html # We only allow /index.shtml to be indexed! User-agent: * Disallow: / Disallow: /cgi-bin Disallow: /dcp Disallow: /hillgc # Disallow: /index.shtml Disallow: /jcp Disallow: /lcp Disallow: /mearsem Disallow: /mka Disallow: /paonet Disallow: /rpao Disallow: /tdye # Paonet spiders are allowed everywhere. # User-agent: SwishSpider # Disallow: User-agent: HtdigPollyPaonetOrg Disallow: User-agent: mnoGoSearch Disallow: User-agent: LinkChecker Crawl-delay: 1 Disallow: # Wed Nov 16 09:48:31 PST 2005 # Google #User-agent: Googlebot #Disallow: