Experimental Distributed Web Crawling with Python + Gearman - View it on GitHub