A generic concurrent tree crawling algorithm with a sample implementation for website crawling - View it on GitHub