robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html