Search.gov fork of robotex Ruby library to obey robots.txt - View it on GitHub