# This file controls access to our resources by WWW robots and spiders. # Author: Kurt D. Starsinic # Last updated: 07/23/99 # The latest version of the robot exclusion standard can be found on # http://web.nexor.co.uk/mak/doc/robots/norobots.html. # # The file consists of one or more records separated by one or more blank # lines. Each record contains lines of the form # ":". # The field name is case insensitive. # # Comments can be included in file using bourne shell conventions. Lines # containing only a comment do not indicate a record boundary. # # The record starts with one or more User-agent lines, followed by one or more # Disallow lines. Unrecognised headers are ignored. # # The value of User-agent is the name of the robot the record is describing # access policy for. # # If more than one User-agent field is present the record describes an access # policy for more than one robot. # # If the value is '*', the record describes the access policy for any robot # that has not not matched any other records. It is not allowed to have two # such records in robots.txt. # # The value of Disallow specifies a partial URL that is not to be visited. # This can be a full or partial path; any URL that starts with this value will # not be retrieved. Any empty value, indicates that all URLs can be retrieved. # At least one Disallow field needs to be present in a record. # # These rules apply to _all_ robots: User-agent: Mediapartners-Google* Disallow: User-agent: * Disallow: /cgi-bin/ Disallow: /etc/wusage/ Disallow: /TOC. Disallow: /sln/ Disallow: /listarchives/ Disallow: /admin/ Disallow: /usr/people/ Disallow: /baldwin/ Disallow: /conventions/ Disallow: /client/ Disallow: /org/ Disallow: /passport/ Disallow: /designlink/ Disallow: /summer98/ Disallow: /175/ Disallow: /news/ Disallow: /community/ Disallow: /explore/ Disallow: /home/ Disallow: /flight_team/ Disallow: /cyberzone/ Disallow: /psd2000/apps/ Disallow: /keystone/apps/ Disallow: /keystone/ Disallow: /keystone/publications/journals2/ Disallow: /conference/ Disallow: /members/ Disallow: /earthcam/ Disallow: /ome/ Disallow: /ome/place/ Disallow: /wright/weather/ Disallow: /wright/weather/registration/ Disallow: /learn/case-files/carver/ Disallow: /ariel/ Disallow: /scholarship/ Disallow: /astc10/ Disallow: /contest/ Disallow: /ldw/ Disallow: /franklinawards/ Disallow: /hawks11/ Disallow: /kje/ Disallow: /change2/ Disallow: /strategic-plan/ Disallow: /TornadoAlley/ Disallow: /exhibitions/scrolls/ Disallow: /exhibitions/ Disallow: /group/ Disallow: /Camp-In/Metrokids Disallow: /discovery-camp Disallow: /summer-camp Disallow: /super-summer Disallow: /VF Disallow: /vf Disallow: /valley-forge Disallow: /VF-camp Disallow: /vf-camp Disallow: /spyexhibit Disallow: /fieldtrip Disallow: /Benefactorboardtoolkit Disallow: /challenge