A curated list of the most common and most interesting robots.txt disallowed directories. - View it on GitHub