New best story on Hacker News: Google’s robots.txt parser is now open source

Google’s robots.txt parser is now open source
425 by dankohn1 | 128 comments on Hacker News.