Waxy.org is the sandbox of Andy Baio, a journalist/programmer living in Portland, Oregon. I'm the CTO of Kickstarter, created Upcoming.org, and some other stuff too.

Contact Me: log@waxy.org or waxpancake on AIM

The Forbidden Web

Posted May 9, 2002

Unethical idea of the day: 'The Forbidden Web,' a search engine that only indexes files disallowed by robots.txt files. For example, CNN's robots.txt file asks search engines to avoid their transcripts, jobs, website statistics, and development directories. The Forbidden Web would index only those forbidden (and often intriguing) directories. Evil, isn't it?

A glance at the robots.txt files on some popular sites: New York Times, Google, Hotwired, eBay, Slashdot, Verisuck, Kuro5hin, Filepile, ZDNet, Epinions, IMDB, BBC, IBM, USA Today, Jakob Neilsen.

You can search Google for more robots.txt files.

3 Comments (Add Yours)

May 9, 2002
10:48 AM

mat wrote:

wow. that IS a great idea.

May 9, 2002
3:36 PM

jkottke wrote:

I wondered about the same thing back when I had to modify my robots.txt file to keep the Wayback Machine from downloading my entire site *6 times a day* (post-9/11). I bet there are quite a few bots out there that specifically look for items forbidden by the robots.txt file. To what end, I don't know.

Sep 9, 2003
12:40 AM

Reform Blog.com wrote:

I actually noticed that google was robots.txt less for a while a few weeks ago

Waxy Links

November 20, 2009

Regretsy gets a book deal — the anonymous author turned out to be April Winchell, collector of audio oddities

Google Chrome OS Demo — a world without a local filesystem and apps; also, the Chrome UI concept video (via)

Patrick Moberg's Internet Vices — funny, Tumblr feels more like beer than wine to me

Charlotte Gainsbourg and Beck's "Heaven Can Wait" — Keith Schofield's surreal video and insane treatment were inspired by FFFFOUND and Reddit, but maybe too explicitly (via)

November 19, 2009

YouTube adds machine-translated automatic captions — starting with some partner channels, but auto-timing is available to everyone today

Microsoft tries to patent Edward Tufte's sparklines — they were recently added to Excel

Leonard Lin's Retweet Avatars for Greasemonkey — a subtle change, but a big improvement

Web-ops god John Allspaw leaves Flickr to join Etsy — he's the last of the original Ludicorp team to go (via)

November 18, 2009

Laptop Steering Wheel Desk — don't miss the product photos

Interview with Ralph Eggleston, Pixar's production designer on WALL-E — from last February, but new to me; I didn't know the Axiom had three passenger classes

NSFW: Animated pixel-art video for Flair's "Trucker's Delight" — warning: very offensive and sexist, but the attention to 16-bit detail by director Jérémie Perin is incredible

NY Observer on Anil Dash's new government 2.0 incubator project — Expert Labs debuted at Web 2.0 today, funded with a $500k grant from the MacArthur Foundation

November 17, 2009

Google's Dan Morrill explains how the Droid autofocus breaks every 24.5 days — this gets second-place for quirkiest Android bug (via)

Conan O'Brien and Andy Richter on Zach Galifianakis' Between Two Ferns — his style of comedy usually makes me uncomfortable, but this made me laugh

The Pirate Bay shuts down their tracker for good — they're switching to DHT instead

November 16, 2009

How Darren at Link Machine Go found Belle de Jour's identity five years ago — Brooke was part of the early UK blog scene

ICU64, real-time visualization of Commodore 64 memory — the developer also posted videos of Paradroid and Boulder Dash (via)

Russell Davies on pretending and "barely games" — his SAP prototype looks like great ambient fun (via)

NYT Magazine on the indie gaming movement — nothing new here, but good overview with a wonderful closing anecdote from Cactus

Tim O'Reilly on the pending War for the Web — "more than that, it's a war against the web as an interoperable platform"

November 14, 2009

Jason Scott rounds up Geocities' top 10 most popular MIDI files — along with a torrent with 51,000 MIDIs rescued by Archive Team

Matt Haughey on the discovery of his brain tumor, treatment, and the Internet's response — there were about 1,000 #mathowielove tweets in 24 hours

Belle de Jour reveals herself after six year of anonymity — only six people in the world knew, she only told her parents yesterday (via)

Paul F. Tompkins debates comedy ethics with Improv Everywhere's Charlie Todd — great discussion, and it's hard not to see where both are coming from (via)

November 13, 2009

Rogue Amoeba stops iPhone app development after App Store idiocy — I'm with Marco, the only fix is allowing external apps, but it's unlikely (via)

Numb3rs on IRC — "Luckily, I speak l33t."

Prank War 8: The Skydiving Prank — hard to say if life-threatening situations are funnier than public humiliation

301 Works, Internet Archive works to preserve URL shortener data — the shorteners will provide regular backups and hand over data on closure, though TinyURL's conspicuously missing

November 12, 2009

Quizipedia — simple game with trivia scraped from Wikipedia entries

Kill Screen, funding a new art magazine about videogames — sounds like the English analogue of Amusement I was hoping for

The Forbidden Web

3 Comments (Add Yours)

Leave a comment