Waxy.org is the sandbox of Andy Baio, a journalist/programmer living in Portland, Oregon. I'm the CTO of Kickstarter, created Upcoming.org, and some other stuff too.

Contact Me: log@waxy.org or waxpancake on AIM

The Forbidden Web

Posted May 9, 2002

Unethical idea of the day: 'The Forbidden Web,' a search engine that only indexes files disallowed by robots.txt files. For example, CNN's robots.txt file asks search engines to avoid their transcripts, jobs, website statistics, and development directories. The Forbidden Web would index only those forbidden (and often intriguing) directories. Evil, isn't it?

A glance at the robots.txt files on some popular sites: New York Times, Google, Hotwired, eBay, Slashdot, Verisuck, Kuro5hin, Filepile, ZDNet, Epinions, IMDB, BBC, IBM, USA Today, Jakob Neilsen.

You can search Google for more robots.txt files.

3 Comments (Add Yours)

May 9, 2002
10:48 AM

mat wrote:

wow. that IS a great idea.

May 9, 2002
3:36 PM

jkottke wrote:

I wondered about the same thing back when I had to modify my robots.txt file to keep the Wayback Machine from downloading my entire site *6 times a day* (post-9/11). I bet there are quite a few bots out there that specifically look for items forbidden by the robots.txt file. To what end, I don't know.

Sep 9, 2003
12:40 AM

Reform Blog.com wrote:

I actually noticed that google was robots.txt less for a while a few weeks ago

Waxy Links

November 7, 2009

NYT visualizes the unemployment rate for different demographics — 48.5% of young black men without a high school degree; 3.6% of college-educated white women over 25

November 6, 2009

Another World level ported to Javascript — in other emulation news, a NES and Gameboy emulator in JS and SNES9x ported to Flash (via)

Blocktronics' ANSI art tribute to RaDMaN — powered by Viewtronics, Peter Nitsch's gorgeous new Flash 10 ANSI viewer (via)

Aaron Straup-Cope leaves Flickr, joins Stamen Design — one of my favorite geeks joins one of my favorite companies

Unreal Engine 3 development kit now free for non-commercial use — huge announcement, along with the recent free release of Unity Indie

The Big Picture's series on Martian landscapes — Kai's Power Tools in real-life (via)

November 5, 2009

Preview of McSweeney's Panorama, their one-shot newspaper — as expected, looks incredibly great (via)

The Grant-Pattishall Award — congrats, Kellan! (via)

Birdhouse for Your Soul — Greg Knauss finds one small piece of the historical web

Google open-sources Closure Tools — JS compiler, along with Google's huge widget library (via)

Video montage of actors speaking the movie's title — great comments with some missed opportunities; "You talkin' to me? You talkin' to The Taxi Driver?"

The Morning News' Cloud of Atlases — impossible to guess, but look at all the pretty colors

American Airlines fires UX designer for explaining why their UX isn't great — a lapse of judgment from both American Airlines and an employee who cared too much

November 4, 2009

Overheating, photo series of gadgets thrown through walls — from issue 6 of Amusement, the incredible French gaming culture magazine (via)

Ricardo Autobahn's The Golden Age of Video — insane pop culture video mashup

November 3, 2009

The Last Days of Gourmet — sad photo series, reminds me of the dot-com carnage photos

Put This On — first episode of Jesse Thorn and Adam Lisagor's Kickstarter-funded video series on clothing

Jono Bacon's The Art of Community released for free download under CC license — looks fantastic and worth buying (via)

Eric Testroete's papercraft portrait Halloween costume — incredibly creepy, like videogames leaking into the real world (via)

November 2, 2009

Mark Pilgrim's history of the IMG element — told through annotated conversations from 1993 (via)

Every vandalism edit to Nickelback's Wikipedia page — I wonder which edits managed to stay in the longest without detection

November 1, 2009

Mike Pusateri's Halloween costume data collection — for the fifth year, he's collected every costume name; this year, "nothing" spiked to #2

XKCD's movie narrative charts — here's a more serious attempt at Primer's timeline

October 30, 2009

GameCity Squared's 15-Pixel Megamix — extremely minimalist interpretations of 12 different games

October 29, 2009

Lauren McCarthy's Happiness Hat — it measures your smile and stabs you if you're not smiling sufficiently (via)

October 28, 2009

Auto Tune de Nieuws — needs an angry Dutch gorilla

Facebook prank memorializes living person — the Facebook team should allow an email veto, or at least require better documentation (via)

2D Boy's pay-what-you-like World of Goo results wrapup — don't miss the breakdown by OS and country (via)

FreeForm's short film on the Open Internet — impressive set of interviewees, directed by Jesse Dylan of Yes We Can fame

Using Flickr as a paintbrush — coloring overhead maps based on the dominant colors of photos taken on the ground (via)

The Forbidden Web

3 Comments (Add Yours)

Leave a comment