Waxy.org is the sandbox of Andy Baio, a journalist/programmer living in Portland, Oregon. I'm the CTO of Kickstarter, created Upcoming.org, and some other stuff too.

Contact Me: log@waxy.org or waxpancake on AIM

The Forbidden Web

Posted May 9, 2002

Unethical idea of the day: 'The Forbidden Web,' a search engine that only indexes files disallowed by robots.txt files. For example, CNN's robots.txt file asks search engines to avoid their transcripts, jobs, website statistics, and development directories. The Forbidden Web would index only those forbidden (and often intriguing) directories. Evil, isn't it?

A glance at the robots.txt files on some popular sites: New York Times, Google, Hotwired, eBay, Slashdot, Verisuck, Kuro5hin, Filepile, ZDNet, Epinions, IMDB, BBC, IBM, USA Today, Jakob Neilsen.

You can search Google for more robots.txt files.

3 Comments (Add Yours)

May 9, 2002
10:48 AM

mat wrote:

wow. that IS a great idea.

May 9, 2002
3:36 PM

jkottke wrote:

I wondered about the same thing back when I had to modify my robots.txt file to keep the Wayback Machine from downloading my entire site *6 times a day* (post-9/11). I bet there are quite a few bots out there that specifically look for items forbidden by the robots.txt file. To what end, I don't know.

Sep 9, 2003
12:40 AM

Reform Blog.com wrote:

I actually noticed that google was robots.txt less for a while a few weeks ago

Waxy Links

March 17, 2010

Crowdsourced demographic study of Chatroulette — the info was gathered by Hacker News users

March 16, 2010

Progress Wars — countless hours of fun

March 15, 2010

Piano Improvisation on Chat Roulette — amazing how much creativity the site's inspiring (via)

March 12, 2010

8-Bit Austin — I think I'll use this map to get to Datapop 2010

Spritely, jQuery plugin for sprite and background animation — see also: gameQuery

March 11, 2010

Trololololololo Shreds — some context (via)

Preview of Sword & Sworcery EP for the iPhone — looks unlike anything I've ever seen

Sitby.us — essential iPhone-optimized site for SXSWi session planning

Danc on the release of Ribbon Hero — turning Microsoft Office into a game, with competition against your friends (via)

March 10, 2010

"Play" by David Kaplan and Eric Zimmerman — avatars as Russian nested dolls (via)

Chatroulette Map — I think I'd rather not know, thanks (via)

Steamshovel Harry — not sure how I missed this one last year, metagaming with music by Brad Sucks

El Fin Del Mundo by Alberto González Vázquez — there's so much I love about this, I can't quantify it all (via)

March 9, 2010

Wired Reread, blogging the best ads from '90s-era Wired — also, the complete SPIN archives are on Google Books

Academy Award Winning Movie Trailer — related: McSweeney's categories for the meta-awards (via)

Chris Parnell and Andy Samberg perform Lazy Sunday live — for the first time, backed by The Roots

Adam Savage's pursuit of the perfect Blade Runner gun replica — related: his quest for the perfect replica Maltese Falcon and dodo skeleton

The Panic Status Board — the instant feedback made work more game-like

March 8, 2010

Valve ports game library and Steam service to Mac — Portal 2 will be released for Mac simultaneously with PC, along with "all of our future games"

Maciej Ceglowski on the discovery, loss, and rediscovery of the cure for scurvy — fascinating story of bad science and the unintended effects of new information

March 7, 2010

8-Bit NYC, Brett Camper's videogame map of New York — he's using Kickstarter to expand to 15 other cities worldwide

Sleep Is Death, Jason Rohrer's new conversational two-player game — watch the slideshow for details; I just wish it was on the web instead

Obama appoints Edward Tufte to advise on stimulus transparency — "Maybe I'll learn something."

PS22 Chorus sings Phoenix's Lisztomania — I love how expressive they are

Echo Nest and SCHED's guide to SXSW Music — very nicely done, uses Echo Nest's recommendation engine

GameInformer's Portal 2 exclusive cover story — scans, since it's not on GameInformer's site yet; Valve hired the TAG: The Power of Paint team right out of Digipen

March 5, 2010

Cal Henderson on gaming probability in World of Warcraft — he's collected 118 pets, some of which only drop 1 in 10,000 attempts

March 4, 2010

LiveJournal rewrites outbound links with affiliate codes — looks like the regex was a bit greedy

NYT on Chinese "human-flesh search engines" — very similar to the H+ article on the topic from last year

YouTube launches auto-captioning for all videos — a free, automated audio transcription service based on YouTube should be viable now

The Forbidden Web

3 Comments (Add Yours)

Leave a comment