Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls. - View it on GitHub