Multi-threaded Web crawler with support for custom fetching and persisting logic - View it on GitHub