Scans robots.txt files and logs the Disallow - used for some research - View it on GitHub