September 21, 2007 Fun with Robots.txt, a survey of 4.6 million domains — if you like that, try Andrew’s Fun with HTTP Headers survey from 2005 #
