# robots.txt
User-agent: Googlebot
User-agent: MSNBot
User-agent: Slurp
User-agent: Teoma
User-agent: Gigabot
User-agent: Scrubby
User-agent: Robozilla
User-agent: Nutch
User-agent: ia_archiver
User-agent: baiduspider
Crawl-delay: 5
Disallow: /portal/cakes_admin/
Disallow: /portal/cache/
Disallow: /portal/cakes_handlers/
Disallow: /portal/public/avatars/