A short and sweet Python web crawler using Redis as the process queue, seen set and Memcache style rate limiter for robots.txt - View it on GitHub