This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster. - View it on GitHub