A "robots.txt" parsing and querying library for .NET - View it on GitHub