Module
robotparser

robotparser.py

Copyright (C) 2000 Bastian Kleineidam

You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2

The robots.txt Exclusion Protocol is implemented as specified in http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html