A python script to check if URLs are allowed or disallowed by a robots.txt file. - View it on GitHub