A more complete robots.txt parser for Python - View it on GitHub