A native Rust port of Google's robots.txt parser and matcher C++ library. - View it on GitHub