robots.txt file parsing and checking for R - View it on GitHub