User-agent: * Disallow: /cgi-bin/ # beware of overloading because of cycles Disallow: /tmp/ # just in case Disallow: /images/ # no need to index images User-agent: ApacheBench/* Disallow: *