Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls - View it on GitHub