# /robots.txt file for http://webcrawler.com/ # mail jason@idmstudio.co.za for constructive criticism User-agent: * Disallow: /_mmServerScripts Disallow: /_notes Disallow: /cart Disallow: /conf Disallow: /Connections Disallow: /css Disallow: /db Disallow: /docs Disallow: /extras Disallow: /gallery Disallow: /hooks Disallow: /images Disallow: /includes Disallow: /info Disallow: /js Disallow: /manager Disallow: /myadmin Disallow: /mymailer Disallow: /nl Disallow: /Scripts Disallow: /sessions Disallow: /shopper Disallow: /SpryAssets Disallow: /stats Disallow: /generator Sitemap: http://www.scubaworld.co.za/sitemap.xml