Holds scrapy, wget, python, etc. code (probably not nutch/tika) to implement DIG web crawlers - View it on GitHub