Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues. - View it on GitHub