Rust crate for configurable parallel web crawling, designed to crawl for content - View it on GitHub