Apache Nutch is an extensible and scalable web crawler - View it on GitHub