The robots.txt exclusion protocol implementation for Go language #golang - View it on GitHub