Guix will hold its first online conference November 22nd. Propose a talk before November 6th. Learn more!

perl-www-robotrules 6.02 Perl database of robots.txt-derived permissions

The WWW::RobotRules module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <>. Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site.