Scrapy middleware which allows to crawl only new content - View it on GitHub