OpenACS

jeffd
committed
MAIN:jeffd:20040426075435
on 26 Apr 04
robots.txt had UserAgent rather than User-agent for the key, added acs-lang, acs-admin, acs-service-contract, workflow, test, and notificati… Show more
robots.txt had UserAgent rather than User-agent for the key, added acs-lang, acs-admin, acs-service-contract, workflow, test, and notifications to the list of disallows.

Show less

openacs-4/www/robots.txt (+18 -0)
1 1 User-agent: *
2 2 Disallow: /doc/
3 3 Disallow: /api-doc/
  4 Disallow: /register
4 5 Disallow: /register/
5 6 Disallow: /comments/
6 7 Disallow: /notifications/
7 8 Disallow: /test/
8 9 Disallow: /workflow/
9 10 Disallow: /acs-service-contract/
10 11 Disallow: /acs-lang/
11 12 Disallow: /acs-admin/
12 13 Disallow: /RPC2/
13 14
  15 # googlebot knows wildcards so block all register
  16 User-agent: googlebot
  17 Disallow: /doc/
  18 Disallow: /api-doc/
  19 Disallow: /register
  20 Disallow: /register/
  21 Disallow: /*/register
  22 Disallow: /*/register/
  23 Disallow: /comments/
  24 Disallow: /notifications/
  25 Disallow: /test/
  26 Disallow: /workflow/
  27 Disallow: /acs-service-contract/
  28 Disallow: /acs-lang/
  29 Disallow: /acs-admin/
  30 Disallow: /RPC2/
  31