I needed a serious web crawler for search engine applications. This is it. - View it on GitHub