Determine if a page may be crawled from robots.txt, robots meta tags and robot headers - View it on GitHub