This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not. - View it on GitHub