This class provides a set of methods to read, parse and answer questions about a single robots.txt file.
There are no base classes.
There are no implemented interfaces.
There are no attributes in this class.
can_fetch(useragent, url)
using the parsed robots.txt decide if useragent can fetch url
modified()
Sets the time the robots.txt file was last fetched to the current time.
mtime()
Returns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to check for new robots.txt files periodically.
parse(lines)
parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines.
read()
Reads the robots.txt URL and feeds it to the parser.
set_url(url)
Sets the URL referring to a robots.txt file.
There are no known subclasses.