I've updated my spreadsheet with all the current available data, eight years of data tracking the online distribution of every Oscar-nominated film since 2003. I've added this year's 34 nominated films to the list, a total of 245 films. (Read about my methodology at the end of the entry.)

View or download all the data below, including a second sheet with some interesting aggregate statistics. As always, I'll keep it updated until the Oscar broadcast. (And let me know if you find any mistakes.)

View full-size on Google Spreadsheets.

Download: Excel (with formulas) or CSV

Findings

Since 2003, I've tracked the online distribution of Oscar screeners, and every year, the piracy scene manages to release nearly every film by nomination day. Last year, all but three films were leaked in DVD quality by nomination day.

The tide may be turning. There's still a month out before the Academy Awards, but so far, fewer Oscar screeners leaked online this year — only 14 out of 34 nominated films, the lowest percentage ever. And they're taking twice as long to leak — a median 21 days after theatrical release, up from 11 days the previous year.

It's not limited to screeners, either. Camcorder and telesync releases dropped this year. Even the percentage of retail DVD rips has dropped, though this will likely shift before the broadcast. In the chart below, you can see the percentage of films that were released in each format. (For example, 21% of this year's films had a cam release and 44% had a retail DVD leak.)

And the R5 DVD releases that dominated previous year's Oscars is now mostly dead. I'm guessing the studios are moving away from the early distribution of R5 DVDs entirely.

But why the shift this year? Are studios doing a better job protecting screeners and intimidating Academy members? Or was this year's crop of films too boring for pirates to bother with? I can't tell if this is a scene-wide trend or localized to the Oscars only. If you have access to historical data tracking scene releases, get in touch.

And if you have any theories or inside information, leave a comment.

Other fun facts:

• Academy members received screeners for 30 of the 34 nominated films.
• The Avatar screener was the last to be received by Academy members (Ken Rudolph received his on January 15). Amazingly, it hasn't leaked online yet. February 4: It leaked today.
• The Hurt Locker and The Young Victoria were both leaked online in DVD format over six months before their theatrical release.
• As far as I can tell, The Secret of Kells is the first film since I started tracking to be nominated without a U.S. theatrical release. It's currently slated to come out in March.

Methodology

As usual, I included the feature films in every category except documentary and foreign films (even makeup and costume design). I used Yahoo! Movies for US release dates, always using the first available date, even if it was a limited release. Cam, telesync, R5, and screener leak dates were taken from VCD Quality. I used the first leak date, with the exception of unviewable or incomplete nuked releases. Finally, the official screener dates came from Academy member Ken Rudolph, who lists the date he receives every screener on his personal homepage. Thanks again, Ken!

For previous years, see 2004, 2005, 2007, 2008 (part 1 and part 2), and 2009.

• Multitrack Analysis of She's Leaving Home
• Multitrack Analysis of A Day In the Life
• Multitrack Analysis of Come Together

Unfortunately, I don't have any information about the source. In the "Come Together" one, they mention one of the narrators is named "Steve." Beyond that, I haven't had any luck finding where they came from. Can anyone identify them? I'd love, love, love to hear more.

Update: It's a BBC Radio 6 program called Record Producers: The Extended Cut, hosted by Richard Allinson and Steve Levine, that aired last month. Unfortunately, the original BBC broadcast is no longer available on their site.

Working with these guys was an absolute dream. Their creativity and dedication transcends the original concept, creating something that pays tribute to Miles' seminal work while pushing the boundaries of the genre.

I know there are jazz purists out there that hate the idea of anyone interpreting a jazz masterpiece in this way, but to them, I'd only ask that you listen to it first before making a judgment. Virt responded to one naysayer in the comments on my original post:

Way I see it, chiptunes can either be a punishingly difficult artistic medium we happened to grow up with, or a tired retro fashion statement. Our goal was to stick to the former, pushing the limitations hard, building on our capacity for expression using the most basic sounds. There could be no better challenge, Andy thought, than one of the most expressive jazz albums of all time, one that has inspired us all.

So, you see, I'm not the least bit embarrassed by our work. In fact, I think you might be short-changing "the masters of jazz," who I believe would be grinning ear to ear right now. They were ALL ABOUT mastering unusual techniques and expressing within a framework. That's the whole point of Kind of Blue. The parallels to our own medium were dead obvious, and I got the same rush of perverse glee that the original ensemble must have felt 50 years ago, locking myself in a cell and playing between the metal bars.

I hope, if you still can't enjoy the sound of the album itself, you might at least be less quick to dismiss it, given this perspective. It's not a parade, it's a love letter in our own weird handwriting.

So, thanks for listening. (Oh, and bonus points to anyone who can identify all the quotes and references in the album.)

I'm happy to say that Code Rush is now released under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 license. Winton and his colleague John Koten set up a dedicated homepage for the film, with links to stream or download the film in various formats.

They're encouraging everyone to use the documentary in new ways, remixing or reusing the footage for any non-commercial use. In particular, I'd imagine the Mozilla Foundation should be very happy that they can finally use this historic footage of their origins.

Thanks to the new license, I'm able to put my annotated version of the film back up on Viddler. I've embedded it below.

Best of all, David Winton's announced that they're planning on digitizing the original interview footage and making them available. "We are still working to get our hands on a digital Beta deck to digitize the original dailies, but hope to get up and running in a couple months." If you can help them out, get in touch.

Update (August 6): I just discovered that unreleased footage from the documentary is being added to Archive.org.

I've been on the board for the last 10 months, helping find the development team and providing some guidance on tech, design, and community issues. And in the last year, watching the site evolve was an amazing experience, from an idea to a website with the potential to change the way things are made.

Since our launch ten weeks ago, over $250,000 has been pledged to make everything from books, magazines, albums (and album reissues), plays, films, art projects, zombie iPhone apps, and more. (Not to mention, my own Kind of Bloop album.) And keep in mind, the site's still invite-only!

Getting people to give you money is tricky, but I think we've hit on a formula for success:

• All-or-nothing. Projects are only successful if they reach the fundraising goal by the deadline, otherwise nobody pays. This limits risk for both backers and project creators, who don't have to worry about committing money and time to a failed project.
• Rewards. We strongly emphasize the importance of crafting good rewards, which makes Kickstarter more like commerce than altruism. We support multiple tiers of rewards from $1 to $10,000, limits for each, and tools for creators to contact each tier group independently.
• Publishing. A simple and powerful reward is access to exclusive updates during a project's funding and development, creating a powerful connection between the audience and project. As a result, we offer publishing tools for public or private updates, including hosted media and update notifications.

These mechanisms and constraints allow Kickstarter to not just fund projects, but test ideas, engage with an audience, and pre-sell your work without risk.

Earlier this month, I spoke at the Guardian Activate Summit in London about the power of play and applying game mechanics to non-games — difficult problems like environmental change, political activism, and fundraising. Kickstarter turns fundraising into a social game, where people have to work together within a time limit to reach a common goal. Already, we're seeing that projects develop their own viral momentum... Once a project hits 25% of its goal, success is almost guaranteed. (94% of projects that hit that mark eventually hit their goal.)

I look forward to pushing Kickstarter further in that direction, and build a platform flexible enough to do more than help artists raise money for themselves. I'd love to see more people use Kickstarter for commissioned works like Kind of Bloop, collecting the money to pay someone to make or do something you want to exist. Or anywhere you need to gauge market demand, like throwing parties without the risk of losing your shirt in ticket sales.

Soon, we'll be opening the site up to anyone who wants to make a project — and that's when things get really interesting. In the meantime, you can get an invite from an existing member, or sign up to get notified when Kickstarter opens to the public.

Like Jon Haddock's porn sans people, these photos are banal out of context. Only someone familiar with the original memes would sense something's amiss, like the set of a play waiting for the actors to stumble into history.

Can you name all 22 23? (Click any image for the answer.)

Ever since Kickstarter launched, I've been trying to come up with a great project for it that plays to its strengths... I like to describe it as a site that lets other people pre-order your dreams — an easy way to get the people you know to fund your ideas into reality.

With that in mind, I just launched a project I've been dreaming about for years. The idea is Kind of Bloop, an 8-bit tribute to Miles Davis's Kind of Blue, one of my favorite albums of all-time. I've always wondered what chiptune jazz covers would sound like. What would the jazz masters sound like on a Nintendo Entertainment System? Coltrane on a C-64? Mingus on Amiga?

I've researched the topic quite a bit, and was only able to find four jazz covers ever released — ast0r's version of Coltrane's Giant Steps and Charlie Parker's Confirmation, Sergeeo's own Giant Steps cover, and Bun's version of Coltrane's My Favorite Things. (If you know more, please let me know!)

So I asked ast0r and sergeeo, along with three incredible chiptune artists (Virt, Shnabubula, and Disasterpeace), to collaborate on a track-by-track remake of the album. I'm raising the money to legally release the album, pay the royalties¹, print a very limited run of CDs for Kickstarter backers only, and pay the artists for their hard work on these very challenging songs.

Read more about the project, and back it if you want to make it real.²

Update: We hit our $2,000 goal in four hours, so this project's definitely on! That doesn't mean it's over, though... Anyone can still give money for the download or limited-edition CD. But I'm not planning on selling the album after the August 1 deadline, so pledge now if you want a copy.

Footnotes

1. This is my first time licensing music, and I'm frustrated that there's no free, legal way to release this album for free download when it's done. By law, you're legally required to pay royalties for every download, whether or not you charge for it. Wouldn't a percentage of revenue make more sense?

2. Some people seem to misunderstand what Kickstarter's for, expecting it to work like Kiva, where there's a pool of investors waiting for neat projects to throw their money into. In reality, I'd expect very, very few projects to be backed by random people stumbling on it from the Kickstarter website. It hinges on your own social network, your ability to promote your project, and the demand for what you're offering. So if your project fails, it's most likely because there wasn't enough interest from the people you know.

From: jenifergoodluck (Your Big Fool) <cynthiawilliam5@yahoo.com>
Reply-to: jenifer.dagba@yahoo.com
Date: Mon, May 4, 2009 at 6:11 AM
Subject: Hello My Dear one

Hello My Dear one

How are you and how is your work? i hope that all is well with you, My name is miss Jenifer , i know that you may be suprise how i get your email, i got your email today when i was browsing looking for honest partner,then i feel to drop this few line to you , and i will like you to contact me through my email so that we can know each other and exchange our pictures, and we maybecome partner.

Remember the distance does not matter what matters is the love we share with each other. i am waiting to hear from you soon.

kiss regards Miss Jenifer

About an hour later, I received a very unusual followup.

From: jenifergoodluck (Your Big Fool) <cynthiawilliam5@yahoo.com>
Reply-to: jenifer.dagba@yahoo.com
Date: Mon, May 4, 2009 at 6:11 AM
Subject: You Owe Me

Since you haven't fallen for my stupid scam letter let me go ahead and be up front with you.

Because I am a Nigerian, you owe me something. The fact that my decadent forefathers sold their neighbors and relatives into slavery means that you owe me a lot of money, especially if you are white. I will accept $1000 USD from you per month for the next 12 months. That will settle your debt towards me that was created by our forefathers.

Moreover, it is imperative that you begin to acknowledge my inherited right to steal and be corrupt without oppression from anybody's legal system. I am entitled to instant riches at the expense of everyone outside West Africa.

This starts with you, my friend, so start paying up now by Western Union.

As much as I'd like to think Jenifer had a nervous breakdown within the hour, it's clear that it's a different author. The writing style is completely different and the scammer's from Senegal, not Nigeria.

I'm guessing an angry recipient hacked her Yahoo! Mail account and sent out the second message to discredit her. Any other theories? I replied to the email to get more details, but I don't expect a response.

Kickstarter aims to let creative people of all kinds -- journalists, artists, musicians, game developers, entrepreneurs, bloggers -- raise money for their projects by connecting directly with fans, who receive exclusive access and rewards in exchange for their patronage. Like Josh Freese and Jill Sobule, the site allows creators to have multiple tiers of rewards (e.g. $20 for the book, $50 for signed copy) with optional limits for each.

The model is simple: a project creator sets a fundraising goal, deadline, and an optional set of rewards for backers. If the goal's reached by the deadline, then everyone's charged via Amazon Payments and the backers get their goodies. If the goal's not reached, nobody's charged. It's all or nothing.

If you want to raise money to build an iPhone app, make a run of t-shirts, or print a book, you can do it with absolutely no risk or up-front costs. If there's enough demand for your idea, you'll be able to sell every copy before you've spent a dime.

Kickstarter also offers publishing tools, where creators can post project updates with audio and video, either publicly or for backers only. For projects without a physical reward, exclusive updates could be a great incentive for people to get involved. Check out this project for a good example.

Anyway, I'm thrilled to see what people come up with! For now, anyone can back projects, but you'll need a Kickstarter invite to be able to create your own project. (You can get an invite from an existing member, or sign up to get notified when Kickstarter opens to the public.)

The nominations for the 13th Annual Webby Awards are in, and browsing the list, I'm a little surprised at how much it's grown. I remember the novelty of the first ceremony at Bimbo's back in 1997, with its quirky five-word speeches and humble 15 categories.

I was curious to see the growth trend, so I tallied up the total number of categories on their official site. In the last five years, we've seen a 330% increase in new categories to a total of 129 today. In the chart below you can see the gradual rise during the dot-com era and brief reduction after the bust, only to swell along with the Web 2.0 movement. In 2005, with the introduction of the new Mobile, Advertising, and Film award types, the number of categories more than doubled to 63 and continued to expand every year since.

With so many categories, you'd think that their business model hinged on getting as many entries as possible... Which, of course, it does. Submitting an entry for Webby consideration costs $275 for the Website, Mobile, and Advertising categories, while the Film categories costs $195.

All of this reminds me of Cool Site of the Day, a former web mainstay that's long since drifted into irrelevance. Once they started taking cash for consideration, the award became less meaningful and the picks were less interesting because of it.

At what point does the Webbys meet the same fate as CSOTD, where the only people who care about the awards are the nominees themselves?

Background

After Del.icio.us founder Joshua Schachter's article about URL shorteners was posted on AllThingsD, he asked on Twitter, "What the hell is this?" Danny Sullivan replied, "It's a compliment. AllThingsD liked your shortener article enough to feature you on their home page." Joshua responded, "It's just very unclear to me where that came from, who wrote it, why they are showing ads on it, etc."

Sunday, 3:30pm
What Do I Do With Myself, Now that the Economy Has Collapsed?

Lane Becker moderates a lineup of web geeks who started projects during the last bust, with some advice and lessons learned from our past success and failures. I'm very lucky to be on the lineup, along with the wonderful Ben Brown, Michael Sippey, and Jane Mount.

Monday, 7:30pm
The Heather Gold Show
Palmer Events Center, 900 Barton Springs Road

Every year, writer/comedian Heather Gold brings her live, interactive talk show to Austin to interview artists, musicians, coders, and writers around a theme. This year's subject is "Something From Nothing," a loose conversation about inspiration and the creative drive, with CD Baby founder Derek Sivers, Huffduffer creator Jeremy Keith, Adaptive Path founder/Emmett Labs CEO Janice Fraser, singer/songwriter Amber Rubarth, and me! The Heather Gold Show is a small part of the huge Plutopia EFF-Austin party, a three-stage art and music extravaganza featuring Bruce Sterling and Ian McLagan from The Faces, so should be fun. Free admission for SXSW badge holders, $10 for everyone else.

Naturally, I'll be on Twitter and my picks for the show are on Upcoming and Sched.org. If you see me, say hi!

While researching Oscar screeners last month, I stumbled on a remarkable example of online collaboration in China that's completely undiscovered here. In short, a group of dedicated fans of The Economist newsmagazine are translating each weekly issue cover-to-cover, splitting up the work among a team of volunteers, and redistributing the finished translations as complete PDFs for a Chinese audience.

It reminds me of the scanlation movement, in which groups of fans scan, translate, and redistribute manga into another language. But I've never seen it applied to a newspaper or magazine, especially one as high-minded as The Economist.

It's an impressive example of online collaboration with simple tools, a completely non-commercial effort by volunteers interested in spreading knowledge while improving their English skills. In the process, they're taking a political risk in translating controversial articles about their homeland behind the Great Firewall.

hodgman: Did I ever tell you people how much I hate the word "meh"? Nothing announces "I have missed the point" more than that word.

hodgman: It is the essence of blinkered Internet malcontentism. And a rejection of joy. Also: 12 hive mehs in the replies SO FAR

hodgman: By definition, it may mean disinterest (although simple silence would be a more damning and sincere response, in that case)

hodgman: But in use, it almost universally seems to signal: I am just interested enough to make one last joyless, nitpicky swipe and then disappear

wordwill: @hodgman Isn't rejecting joy how one traditionally demonstrates one's superior cool? Though, at the same time, to hell with that.

hodgman: @wordwill yes. It's part of the toxic Internet art of constant callous one upsmanship. And it is a sort of art, but not for me.

Original in English

Portuguese, "O-La-Ri-Lo-Le"

Italian, "Urca Tirulero"

I've been tracking the distribution of Oscar-nominated films every year, culminating with the release of six years of piracy data last year. I've updated those spreadsheets with this year's 26 nominees, for a total of 211 films from the last seven years.

You can view or download all the data below, including a second sheet with some interesting aggregate stats. As always, I'll keep it updated until the Oscar broadcast.

View full-size on Google Spreadsheets.

Download: Excel (with formulas) or CSV

Findings

So, how did they do? Out of 26 nominated films, an incredible 23 films are already available in DVD quality on nomination day, ripped either from the screeners or the retail DVDs. (All 26 were available by February 7.) This is the highest percentage since I started tracking.

Only three films are unavailable — Rachel Getting Married wasn't leaked online in any form, while Changeling is only available as a low-quality telecine transfer and Australia as a terrible quality camcorder recording. (Update: A DVD screener of Australia was leaked on January 23, a retail DVD rip of Changeling on January 31, and finally, the retail DVD of Rachel Getting Married on February 7.)

Other findings:

• Academy members received screeners for at least 20 of the 26 films.


• 25 out of 26 films leaked in some form online, if you include camcorder recordings.


• The average time from the time screeners are received by Academy members to its leak online is 6 days.

Surprisingly, it seems like this year's Oscar movies took longer to leak online than in previous years. If I had to guess, it's because far fewer camcorder copies were released for this year's nominees. This could be because of the theaters cracking down on camcorder recordings, but I suspect it's because fewer nominees were desirable targets this year for cams. (Aside from the obvious blockbusters, like Dark Knight, Kung Fu Panda, and Tropic Thunder.) The chart below shows the median number of days from a movie's US release date to its first leak online.

Last year, one of the interesting findings was how the release of Region 5 DVDs were reducing the prestige of official screener leaks. This year, only four of the nominated films were released as R5s, compared to eight from last year. The numbers are still too small to tell if this is a trend, but it seems like the popularity of the R5 may have peaked in 2007. (Are the studios releasing fewer R5s in general?)

What other trends in the data am I missing? Feel free to chime in with your conclusions or visualizations in the comments.

Methodology

As usual, I included the feature films in every category except documentary and foreign films. I used Yahoo! Movies for US release dates, always using the first available date, even if it was a limited release. Cam, telesync, R5, and screener leak dates were almost universally taken from VCD Quality. I used the first leak date, with the exception of unviewable or incomplete nuked releases. Finally, the official screener dates came from Academy member Ken Rudolph, who lists the date he receives every screener on his personal homepage. Thanks again, Ken!

For previous years, see 2004, 2005, 2007, 2008 (part 1 and part 2).

Update: The screener for Australia was released today, so I added that date to the spreadsheet, along with some missing retail DVD dates from last year's Oscars.

February 3, 2009: Some related links of interest... I was interviewed for Future Tense on American Public Media, talking about this entry. Bruce Lidl looked at leaks in the Foreign and Documentary categories, as well as how quickly HD-quality leaks are happening. Finally, Flowing Data is sponsoring a contest to generate information visualizations from this data.

Last week, I started a new Turk experiment to answer two questions: what do these people look like, and how much does it cost for someone to reveal their face?

Answer #1. This is what Mechanical Turk looks like (click for full-size):

Answer #2. About $0.50.

Results

Here's my original request:

Upload a photo of yourself holding a handwritten sign that says "I Turk for ...", filling out why you turk. For example, "I Turk for Cash," "I Turk for My Kids," "I Turk to Kill Time," or whatever else you like. Be honest, be funny, be whatever you like.

As a good faith gesture, here's my photo.

If you have a webcam, you can simply go to Cameroid to snap a photo from your web browser, download the JPG, and upload it below. (Don't worry if the text is backwards, I can fix that myself.) DON'T provide any identifiable information, like your name or email, since that's a violation of MTurk policy.

The result will be used in a collage that can be found on my personal weblog, http://waxy.org. By uploading your image and accepting payment for the image, you give permission to me, Andy Baio, to use your image in all forms and media for any lawful purposes. (That's just cover-my-ass language. I'm almost certainly only going to restrict it to this one project.) The collage will show up there shortly after the HIT is complete. Thanks, everybody!

I started the task at $.05, but only two people responded in the first 24 hours. (And one of those was Joshua Schachter, who I'd told about the project.) Clearly, that was too low, so I increased it to $.25, receiving only eight submissions in 48 hours. (For reference, all 500 of my Girl Talk tasks were done in about an hour.) Increasing it to $.50 got me 20 more submissions in about 48 hours, after which it started to drop off quickly. I wasn't about to give dollar bills to random people for their photos, so I ended the experiment there. People aren't willing to give up their anonymity for cheap.

The final results: 30 people total — 10 women, 20 men. Almost all were white, mostly in their 20s and 30s. 21 said they turked for money, 9 for fun or boredom.

Thanks for pulling back the curtain, Turkers.

Here's a list of every video I could find. Let me know if I missed any.

Anthrax's Scott Ian, "Madhouse" at Best Buy

"You suck. You're going to have to write easier songs... 20 years ago."

Google's flirted with voice-to-text conversion in the past, with GOOG-411 and their Audio Indexing of political videos on YouTube. But this is the first time they're offering a web-accessible interface for speech conversion, albeit completely undocumented, so I decided to poke around a bit to see what I could find.

Over the last few hours, I've been analyzing the traffic proxied through my network, trying to reverse-engineer it to get to something usable, but I've hit my limits. I'm posting this with the hopes that someone out there can run with it and find out more.

Behind the Scenes

Here's what we know so far: When you first start speaking into the microphone, the app opens a connection to Google's server and starts sending over chunks of audio, almost certainly encoded with the open-source Speex codec.

The waveform image is generated on the phone and displayed along with a "Working" indicator and the adorable "beep-boop" sounds. In the background, a tiny file is being sent as a POST request to http://www.google.com/m/appreq/gmiphone. Here's what the headers look like:

POST /m/appreq/gmiphone HTTP/1.1
User-Agent: Google/0.3.142.951 CFNetwork/339.3 Darwin/9.4.1
Content-Type: application/binary
Content-Length: 271
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Pragma: no-cache
Connection: keep-alive
Connection: keep-alive
Host: www.google.com

The response from Google is an even smaller attachment. These two files are the same for every query, so don't contain any meaningful information.

HTTP/1.1 200 OK
Content-Type: application/binary
Content-Disposition: attachment
Date: Tue, 18 Nov 2008 13:06:53 GMT
X-Content-Type-Options: nosniff
Expires: Tue, 18 Nov 2008 13:06:53 GMT
Cache-Control: private, max-age=0
Content-Length: 114
Server: GFE/1.3

After the audio's sent to Google, they return an HTML page with the results and a second request is triggered, this time a GET request to clients1.google.com with the converted voice-to-text string.

GET /complete/search?client=iphoneapp&hjson=t&types=t
    &spell=t&nav=2&hl=en&q=chicken%20soup HTTP/1.1
User-Agent: Google/0.3.142.951 CFNetwork/339.3 Darwin/9.4.1
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Pragma: no-cache
Connection: keep-alive
Connection: keep-alive
Host: clients1.google.com

The response is an array of search terms in JSON format, for use in search autocompletion.

["chicken soup",[["http://www.chickensoup.com/","Chicken Soup for the Soul",5,""],["http://www.chickensoupforthepetloverssoul.com/","Chicken Soup for the Pet Lover's Soul",5,""],["chicken soup recipe","489,000 results",0,"2"],["chicken soup for the soul","1,470,000 results",0,"3"],["chicken soup dog food","462,000 results",0,"4"],["chicken soup with rice","467,000 results",0,"5"],["chicken soup diet","453,000 results",0,"6"],["chicken soup from scratch","364,000 results",0,"7"],["chicken soup for the soul quotes","398,000 results",0,"8"],["chicken soup crock pot","604,000 results",0,"9"]]]

Help!

Unfortunately, until we can isolate and decode the audio stream, playing with the voice recognition features is out of reach.

Any ideas on cracking this mystery would be hugely appreciated. Anonymity for Google insiders is guaranteed!

Updates

As several commenters figured out, and confirmed to me by Google, the audio is being sent to Google's servers for voice recognition. The two binaries I posted above aren't the actual transmission, and are actually identical for every query, so can be disregarded. Sorry about the red herring.

Gummi Hafsteinsson, product manager for Google's Voice Search, says, "I can confirm that we split the audio down to a smaller byte stream, which is then sent to Google for recognition, but we can't really provide any details beyond that." Responding to my request for a public API, he added, "I appreciate the suggestion to provide voice recognition as a service. Right now we have nothing to announce, but we'll take this feedback as we look at future product ideas."

Also, Chris Messina discovered some secret settings in the application's preferences file, including alternate color schemes and sound sets for "Monkey" and "Chicken." Beep-boop!

Next step: As Paul discovered in the comments, the Legal Notices page says clearly that the app uses the open-source Speex codec for voice encoding. Can anyone capture and decode the audio being sent to Google?

November 19: I rewrote most of this entry to reflect the new information, since it was confusing new readers.

(Credit: Michael Buchino, also available as a shirt)

Well, the official CDs were shipped out last week to everyone who donated more than $10. Inside, as promised, was the official sample list — a one-page insert with every single sample on the album. Steve Heil was the first to scan it and contact me.

Unfortunately, a huge block of printed small-caps text isn't very useful for my kind of fun, so I tried throwing into several OCR engines on WeOCR to turn the image into text. Tesseract gave the best results, but it was still a mess that needed quite a bit of cleanup.

Anyway, here it is. The complete list of all 322 samples in Girl Talk's Feed the Animals, available as a CSV, Excel, or Google Spreadsheets document.

While most political blogs are extremely partisan, their biases aren't immediately obvious to outsiders like me. I wanted to see, at a glance, how conservative or liberal the blogs were without clicking through to every article.

With the help of del.icio.us founder Joshua Schachter, we used a recommendation algorithm to score every blog on Memeorandum based on their linking activity in the last three months. Then I wrote a Greasemonkey script to pull that information out of Google Spreadsheets, and colorize Memeorandum on-the-fly. Left-leaning blogs are blue and right-leaning blogs are red, with darker colors representing strong biases. Check out the screenshot below, and install the Greasemonkey script or standalone Firefox extension to try it yourself.

Note: The colors don't necessarily represent each blogger's personal views or biases. It's a reflection of their linking activity. The algorithm looks at the stories that bloggers linked to before, relative to all other bloggers, and groups them accordingly. People that link to things that only conservatives find interesting will be classified as bright red, even if they are personally moderate or liberal, and vice-versa. The algorithm can't read minds, so don't be offended if you feel misrepresented. It's only looking at the data.

For example, while Nate Silver of FiveThirtyEight may be a Democrat, he has a tendency to link to stories conservative bloggers are discussing slightly more often than liberal bloggers, so he's shaded very slightly red. (Geeks can read on for more details about how this works.)

Of course, this is all very frivolous and has nothing to do with the current campaign. But like Barack Obama's high school basketball footage, it's a little glimpse into the early life of a highly-visible national figure.

In the first part added to YouTube, he posted the portion from the swimsuit competition, prefaced by a brief introduction mentioning the demand for the "88 minutes of Alaska Gold."

Update: The original video was removed, but I managed to save a copy of the relevant footage without Richard's original intro. YouTube's removing every copy of this video, so I'm streaming the clip below from my own server. It won't be removed.

April 28, 2009: Kickstarter is live! I wrote more about the launch here.

Kickstarter aims to let creative people of all kinds — journalists, artists, musicians, game developers, entrepreneurs, bloggers — raise money for their projects by connecting directly with fans, who receive exclusive access and rewards in exchange for their patronage. More than just a fundraising app, Kickstarter's a publishing platform where project creators can communicate with the people that are supporting them. (Think Jill Sobule, A Swarm of Angels, or Sean Tevis.)

I was introduced to founders Charles Adler, Perry Chen, and Yancey Strickler by Caterina Fake back in June, and sealed the deal after a trip to NYC to meet the team. They're a great group of guys with a strong vision, and I feel lucky to be involved.

Ultimately, everybody should be able to support themselves doing what they love using the web, and I think Kickstarter will be a great way to get there. Expect to hear more on Waxy.org as launch day gets closer.

To help them on their way, they're currently looking for a CTO to join the founding team. I've been helping guide some of the technology decisions and building the development team, but we're looking for a passionate and talented person to devote themselves to the project full-time.

If you're interested, drop me an email or IM and I'll introduce you!

Bracing for a good four or five hours of rewinding and writing and rewinding, I remembered that this is The Future! So, instead, I tossed the job over to the global anonymous workforce at Amazon Mechanical Turk instead.

The result: my 36-minute recording was transcribed while I slept, in less than three hours, for a grand total of $15.40.

This is a fraction of the cost/time of any other transcription service online, including the Turk-driven Casting Words, though you potentially sacrifice some quality. In my experience, though, there were virtually no errors.

Here's how to do it yourself, with no programming knowledge required. The instructions below are verbose, but using my template, it shouldn't take you more than five minutes of setup per job.

Now, the good news — In my reply, I asked David if he'd mind being interviewed, and he agreed! He's an accomplished director and producer, the creator of the Big Thinkers series for TechTV, and the cofounder of Winton/duPont Films, located in San Francisco's Presidio.

We had a wonderful conversation about the film, which revealed for the first time that he's planning on not only re-releasing Code Rush digitally, but considering releasing the original outtakes (100 hours of footage) to the public domain on Archive.org.

I wish all my takedown notices were like this! Read on for the full interview, with selected clips from Code Rush, used by permission.

Bandcamp is a free hosted publishing platform for musicians, taking the technical challenge out of setting up a site — transcoding music into different formats, streaming audio, analytics, payment processing, and so on.

Band websites are often pretty bad, hacked together by a friend of the band with Flash and Dreamweaver, or worse, by the record label. There are exceptions, but mostly, it's a sea of Flash intros, popup windows, mystery navigation, and 30-second sound clips.

Bandcamp is trying to change that, giving every album and track its own page with clean URLs and semantic markup, with the accompanying SEO benefits. Even before launch, they're topping Google results for many searches for song titles of participating bands.

As an infoviz geek, I'm particular fond of their analytics and audio visualizations. Detailed stats let bands track recent activity on their songs and albums, including where people are coming from, trend tracking, and which songs were skipped, played partially, or played in full. A number of real-time audio visualizations in Flash are available on each song's page, which can be shared and embedded on other websites.

Like Oddpost, the team's small and nimble — only four people, all splitting engineering and design duties. Co-founder/CTO Shawn Grunberger (also formerly with Oddpost and Yahoo! Mail) and two engineers working from Seattle and Vermont round out the distributed team.

Ethan was kind enough to sit down with me on launch day to talk about their inspiration and process developing Bandcamp.

Fortunately, one unlikely celebrity couple is here to guide us through the brave new world of spread sheets, data banks, and modems. In Computability, an instructional VHS tape from 1984, comedian Steve Allen and actress Jayne Meadows "take us on a light-hearted but detailed tour of the ways a home computer can change your life by simply using the correct software packages to suit your needs."

The video was originally inspired by the couple's Grammy-nominated "Everything You Wanted to Know About Home Computers," a vinyl LP released by Casablanca/Polygram Records in 1983. The LP's completely unavailable, but thanks to Sammy Reed's wonderfully strange podcast, I was able to recreate the full album. (Stream it below or download the 11 MB MP3.)

Last month, Rex Sorgatz asked about collecting metadata on the album for data crunching. After spelunking through Billboard's chart history, that sounded like my idea of a good time.

So I compiled all the data into spreadsheets, used Amazon's Mechanical Turk to collect some additional information, and pulled out a few charts. As always, I've provided CSV downloads for all the data along with the original output from Mechanical Turk, for those interested in experimenting with the platform.

Update (October 30): Here's the official sample list.

At the time, NBC's online coverage was restrictive by today's standards — mostly highlight clips and no live video, delayed until after the events aired on TV, and required a valid credit card to verify residency in the United States.

But that was four years ago! YouTube hadn't launched yet, HD-quality streaming video on Vimeo was three years away, and BitTorrent or HDTV were only popular with early adopters.

This year, it's much improved, albeit with some caveats. NBC's official video is great quality, if you and your computer can stomach Silverlight (unavailable on non-Intel Macs). Their coverage is fantastic, though still tape-delayed. And, because of IOC regulations forbidding international distribution, NBC won't allow you to download, embed, or transcode any videos for your iPod or phone.

Is this availability enough to satiate the pirates, and what does the quality look like compared to 2004? I went poking through Usenet and some public and private BitTorrent trackers to see.

Usenet

Back in 2004, the place to go for illegal Olympic videos wasn't BitTorrent, popular trackers like Suprnova, or mainstream P2P clients. The best coverage, surprisingly, was found in the old-school Usenet binaries. It was a mish-mash of events, skewed heavily towards events with bikini-clad women, Brazilians, or bikini-clad Brazilian women, but other popular events and the opening ceremonies also showed up.

Today, the event coverage in Usenet is just as sporadic, but the quality is dramatically better. Compare the three videos below. The first is a sample from the gymnastics high bar finals from the 2004 games, followed by the same footage of Michael Phelps' win from Saturday's 400m IM final, as seen on NBCOlympics.com and a 720p HDTV rip found in Usenet.

Size Comparison (See Actual Size)

Sample Videos (right-click to download):

• Men's Gymnastics High Bar Finals - Usenet, 2004 (25MB MPEG1)
• Men's Swimming 400m IM Final - NBCOlympics.com, 2008 (5MB MPEG-4)
• Men's Swimming 400m IM Final - Usenet, 2008 (15MB MPEG-4)

Here's the full list of Olympics videos currently up on Usenet, as of this evening:

Olympic Games Opening Ceremony (720p)
Football - Group A - Ivory Coast vs. Argentina Extended Highlights
Football - Group B - Netherlands vs. Nigeria Extended Highlights
Football - Round 1 Highlights
Gymnastics - Men's Qualifying - USA
Shooting - Women's 10m Air Pistol Final
Swimming - Men's 100m Backstroke Semifinals
Swimming - Men's 100m Breaststroke Final
Swimming - Men's 200m Freestyle Semifinals
Swimming - Men's 400m Individual Medley (720p)
Swimming - Men's 4x100m Freestyle Final
Swimming - Women's 100m Backstroke Semifinals
Swimming - Women's 100m Breaststroke Semifinals
Swimming - Women's 100m Butterfly Final
Swimming - Women's 400m Freestyle Final
Volleyball - Women's Preliminaries - China vs. Switzerland

Most of these are in alt.binaries.tv, but some are also posted to alt.binaries.multimedia.sports. I'll update this list at the end of next week.

BitTorrent

But the trend for this year is clear — Usenet passed the torch to BitTorrent.

A quick search on Mininova or BTJunkie returns a huge list of every video found on Usenet, plus dozens more and growing hourly. Beyond public trackers, I've seen extensive activity on several private communities. On one of them, its members compiled a list of every event and were slowly adding their own recordings to create a massive archive of Olympics video.

And this is only Day 4! It'll be interesting to see how much of the Olympics was captured, digitized, and uploaded by the end of the games.

Also interesting: If this chart from Mark Ghuneim is accurate, the thirst for pirated Olympics coverage is greatest in China.

We now have a glimpse into Monday's traffic, thanks to a snapshot provided by Kellan and Rabble's in their talk, Beyond Rest: Building Data Services with XMPP PubSub, presented earlier today at OSCON in Portland:

On July 21, 2008, Friendfeed hit Flickr 2.9 million times to get the latest photos of 45,754 users, of which 6,721 visited Flickr in that 24-hour period, and could have potentially uploaded a photo.

Three million requests for 6,000 updates. Clearly, polling isn't ideal. Don't miss the rest of the slides.

(Also, at its peak, Flickr is currently receiving 60 uploaded photos a second, "roughly 10 times the number of people born on Earth per second.")

With its vibrant oversized photographs and minimalist design, the Boston Globe's The Big Picture weblog launched on June 1 to instant global acclaim. It's designed, programmed, and written by Alan Taylor, an old-school web programmer and blogger, in his spare time while working on community features at Boston.com. (You might know Alan from his popular MegaPenny Project, Amazon Light, or his other projects.)

The idea's simple, but extremely effective. Spend a few minutes with the Iowa floods, the faces of Sudan, or the daily life in Sadr City, and you feel like you've opened a window to another world.

I interviewed Alan about the inspiration for the site, his methodology, and what it's like being a programmer in a journalist's world.

The Big Picture's become an essential read for me, and I totally agree with Jason Kottke when he called it "the best new blog of the year." What inspired it?

Alan Taylor: Lots of things — my parents used to always have Life and National Geographic magazines around the house, I fell in love with the visual storytelling way back then. When I was getting my feet wet in the online journalism world as a developer at msnbc.com, I had the good fortune of working alongside Brian Storm and a few others in MSNBC's photo department, who were just phenomenal as far as selection, editing and presentation.

I wondered why other sites didn't reach that level. Many have by now, but I was still frustrated by the presentation — either far too small, or trapped in click-after-click interfaces that were in Flash or just acted as ad farms.

Independent filmmakers followed the Mozilla team from March 1998 to April 1999, as they worked to open Netscape Communicator's source code to the world, in a last-ditch effort to save the company. The result is an amazing snapshot of computer history, capturing the people that worked on it, the first internal beta test, the moment Jamie Zawinski uploaded the first builds publicly, the launch party, the all-hands meeting announcing the AOL acquisition, and so much more. It aired on PBS nationally in March 2000, the same month as the beginning of the dot-com collapse.

Out-of-print and never released on DVD, the used VHS copies start at $50 on Amazon. Like all the videos I release on Waxy.org, this material is commercially unavailable. If they ever come back into print, or the copyright holders contact me, I'll take them down immediately.

Important Update (September 16): At the request of the the director, I've removed the video from Waxy.org and Viddler. I've interviewed the director about his plans for releasing the film and the unreleased footage.

Update (July 31): The documentary is back online, legally released under a Creative Commons license.

I've done my best to annotate the video, but many people in the film aren't identified. I've left Viddler annotations open to everyone, so if you want to identify the people, places, or notable objects/events/trivia in the film, then please add your inline comments the video! (Or IM/email me and I'll take care of it.)

The video's now offline, but I've saved all the annotations. Thanks to Tman for creating the subtitle file, which can be used in video players like Media Player Classic or VLC, or simply viewed as plain text.

Now go download Firefox 3.0 and help make history!

Interviews and Appearances

• Jamie Zawinski: Left Netscape on April 1, 1999, now the owner of DNA Lounge in San Francisco
• Jim Barksdale, CEO
• Michael Toy
• Jim Roskind
• Tara Hernandez: Now an infrastructure engineer at Pixar
• Scott Collins: Now works on the Slashdot engineering team
• Jeff Weinstein
• Marc Andreessen
• Stuart Parmenter (and his parents)
• Brendan Eich: CTO at Mozilla
• David Readerman, Tech Analyst
• Po Bronson, Wired Magazine
• Kara Swisher, Wall Street Journal
• Gregg Zachary, Wall Street Journal
• Ellen Ullman, Author of Close to the Machine

And, as promised, here's the BitTorrent file for high-resolution copies of all five videos. It's a 3.1GB download with five H.264 encoded MP4 files. (If you only want a single video, use your BitTorrent client to select only the videos you need.) Enjoy!

(Previously: Part 1, Part 2, Part 3, Part 4.)

Interviews:
Robert Lucky (AT&T Bell Labs), Dave Hughes, Kathleen Bonner (Trader, Fidelity), George Hayter (Former Head of Trading, London Stock Exchange), Ben Bagdikian (UC Berkeley), Arthur Miller (Harvard Law School), Forman Brown (songwriter, died in 1996), Tan Chin Nam (Chairman, National Computer Board of Singapore), B.G. Lee (Minister of Trade and Industry, Singapore), Lee Fook Wah, (Assistant Traffic Manager, MRT Singapore), David Assouline (French Activist, now a senator), Mitch Kapor (founder, Lotus), Michael Drennan (Air traffic controller, Dallas-Fort Worth)

It's a whirlwind tour of computing before the Web, with brilliant archival footage and interviews with key players — several of whom passed away since the filming. Jointly produced by WGBH Boston and the BBC, it originally aired in the UK as The Dream Machine before its U.S. premiere in January 1992. Its broadcast was accompanied by a book co-written by the documentary's producer Jon Palfreman.

With the help of Simon Willison, Jesse Legg, and (unofficially) the Portland State University library, we've tracked down and digitized all five parts. This week, I'm uploading them, annotating them with Viddler, and posting them here as streaming Flash video as they're finished. Also, the complete set is available for download as high-quality MP4 downloads via BitTorrent.

Here's the first of the five-part series, The Machine That Changed the World. Enjoy!

Thanks again to Martin Brewer, who also contributed the Horizon show, here are two more episodes from the first season of The Net from 1994.

As part of that research, I've been digging into my original server logs from the Star Wars Kid debacle, five years after I played a major role in what some say is the biggest viral video of all-time.

Be warned, this is more detail than you'll ever want about the origins of the Star Wars Kid meme and how it spread. You don't care about this level of detail, but I'm writing this all down so that I never have to think about it again.

In addition, I've decided to release the first six months of server logs from the meme's spread into the public domain — with dates, times, IP addresses, user agents, and referer information. (Download it below.)

Early Origins

Like I mentioned in my original entry, the video was first released by Ghyslain's schoolmates to Kazaa on April 19, 2003 with the original filename "ghyslain_razaa.wmv." Within three days, it was being passed around in the offices of Raven Software in Madison, Wisconsin, where a game developer named Bryan Dube posted it on his personal website on April 22. Two days later, he created the first Star Wars Kid remix, adding lightsabers and sound effects in a new video titled "TheLastHope.avi."

On April 27, a mostly-NSFW online community called Sensible Erection linked to the video on Bryan's website. Later that evening, an SE user cross-posted it to a private file-sharing community I belong to with the new filename "star_wars_guy.wmv." It quickly became the most popular file on the site, which is where I found it the following day, April 28 at 7:52pm.

On April 29, I renamed it Star_Wars_Kid.wmv and posted it to my site at 4:49pm &mdash inadvertently giving the meme its permanent name. (Yes, I coined the term "Star Wars Kid." It's strange to think it would've been "Star Wars Guy" if I was any lazier.) An hour later, Scott Gowell becomes the first person to link to the video.

From there, for the first week, it spread quickly through news site, blogs and message boards, mostly oriented around technology, gaming, and movies. Throughout the life of the meme, most of the referers are blank, suggesting people were primarily sending the links by email or instant message.

The chart below shows the distinct top-level domains that appeared in the referral logs grouped by day.

It's worth noting that the majority of sites sent less than 10 referers in that first month, and 21% of domains referred only one user. (Note: The chart below is on a logarithmic scale for both axes.)

Mainstream News Coverage

Here's some of the highlights from the mainstream media coverage. The New York Times was the first major paper to report on it, almost a week after I tracked Ghyslain down, Jish and I interviewed him for the first time, and we started the fundraiser.

May 19, New York Times
May 19, Wired News
May 20, Public Radio International's "The World" (radio program)
May 20, Globe and Mail
May 20, National Post
May 23, The Mirror UK

Jun 6, LA Times

Jul 4, The Independent UK
Jul 12, The Age
Jul 23, Wired News
Jul 25, BBC News
Jul 31, NPR w/Tavis Smiley (radio interview)

Aug 21, USA Today, syndicated Associated Press article
Aug 25, NBC's Today Show (TV program)
Aug 26, MSNBC's Countdown (TV program)
Aug 28, USA Today
Aug 30, Seattle P-I

Sep 8, SF Chronicle
Sep 15, Variety
Sep 16, Globe and Mail

Nov 18, CBS Evening News

Statistics

Here's what the Star Wars Kid meme did to my overall traffic. At its peak, I received almost a million pageviews in a single day.

That includes all pageviews on my weblog entries. Isolating only the video downloads from my site, or later redirected to one of the mirrors, gives the following chart.

Download the Data

This file is a subset of the Apache server logs from April 10 to November 26, 2003. It contains every request for my homepage, the original video, the remix video, the mirror redirector script, the donations spreadsheet, and the seven blog entries I made related to Star Wars Kid. I included a couple weeks of activity before I posted the videos so you can determine the baseline traffic I normally received to my homepage.

The file is 158 megabytes &mdash 1.6GB uncompressed — so I'm distributing it with BitTorrent. The data is public domain. If you use it for anything, please drop me a note!

Download: star_wars_kid_logs.zip.torrent

Kasparov is a leader of the Other Russia movement, a loose coalition of activists opposing Vladamir Putin and the current Russian government. Over 700 people showed up for the event in central Moscow, but Kasparov's speech was interrupted when a large phallus-shaped helicopter started buzzing around the room. The Moscow Times attributed the prank to "a couple of pro-Kremlin Young Russia activists."

Warning: Mildly NSFW images and video follow.

Longevity of a Pop Song

One of the trickier questions I've been trying to visualize is how long pop songs are staying on the charts relative to the past. Are they staying on the charts longer than in the past?

In the chart below, I plotted the total number of weeks charted for all 23,924 songs that appeared on the Billboard Hot 100 from 1957 to earlier this year. (In other words, a little dot on the "60" line means there was a song released that week that stayed on the Hot 100 chart for 60 weeks.)

See the heavy dropoff on the 20th week starting in 1991? In an attempt to increase diversity and promote newer artists and songs, Billboard changed their methodology, removing tracks that had been on the Hot 100 for twenty consecutive weeks and slipped below the 50th position. These songs, called "recurrents," were then moved to their own chart in 1991, the Hot 100 Recurrent.

Unfortunately, this shift makes it much harder to compare the last 15 years to the decades before it. In the chart below, I've isolated the effect by only showing songs that reached the top 50.

A couple interesting observations... Looking at the very bottom of the chart, you can see that in the last couple years, it's become very common for a single to appear in the Top 50 and fall out of the Hot 100 within four weeks. Prior to the mid-1990s, this almost never happened.

Also, songs are staying in the Top 50 for far longer than they used to. Unfortunately, I don't have any actual sales numbers to compare to, so it's hard to say if these 30-70 week singles are massive megahits eclipsing the #1 singles of the past, or if it's because the record industry is producing fewer hits than before.

Diversity

Did Billboard's methodology changes in 1991 make the charts more diverse, like they hoped? By looking at the total number of unique songs that have charted yearly, it's clear their changes did nothing to slow the decline.

According to Billboard, the late 1960s were the peak of musical diversity in popular music, with 743 different songs appearing on the 1966 Billboard Top 100. It's fallen consistently since, hitting an all-time low in 2002 with only 295 songs. Since then, it's improved only slightly, with 351 unique songs appearing on last year's Top 100.

One Hit Wonders

I've always thought the 1970s were the decade of the one-hit wonder, but now I have the data to see for sure.

In raw numbers, the 1960s had more one-hit wonders than any other decade, followed closely by the 1950s. But that's not entirely fair since, as we saw earlier, there were simply more unique songs on the 1960s charts. To find out the true numbers, we need to look at the number of one-hit wonders as a percentage of all songs in the Top 100.

This tells a totally different story. The 1960s, 1970s, and 1980s all had about the same ratio of one-hit wonders to hits by more established artists. The big surprise for me is that 1950s, 1990s, and 2000s really seem to be the eras where one-hit wonders dominated the charts.

Joshua Porter was wondering about the longest-charting one-hit wonders of all time. The longest-charting one-hit wonder to hit the #1 spot is Daniel Powter's "Bad Day" from 2006, which stayed on the charts for 32 weeks. The one-hit wonder that stayed at the #1 longest is Anton Karas' "The Third Man Theme" from 1950, which stayed in the #1 position for 11 weeks. Finally, the longest-charting one-hit wonder to appear anywhere in the Top 100 is Duncan Sheik's "Barely Breathing" from 1997, which peaked at #16 but stayed in the top 100 for 55 weeks.

Have any other questions about the data, or done any analysis yourself? I'd love to hear about it.

May 20: Don't miss Mike Frumin's chart of pop longevity,from 1998-2002.

May 21: Using the Whitburn data, Tom Whitwell generated a tag cloud showing the top 100 commonly-used words in song names. Dianne Warren should write a #1 hit called "Love my Baby Blue Heart: A Girl's Night Song."

July 14: Pedro did some additional analysis, including artists with multiple hits in the same week and one-week wonders.

For the last ten years, obsessive record collectors in Usenet have been working on the Whitburn Project — a huge undertaking to preserve and share high-quality recordings of every popular song since the 1890s. To assist their efforts, they've created a spreadsheet of 37,000 songs and 112 columns of raw data, including each song's duration, beats-per-minute, songwriters, label, and week-by-week chart position. It's 25 megs of OCD, and it's awesome.

As far as I know, this is the first time the project and its data have ever been discussed outside of Usenet. Despite its illegality, they've created a wonderful resource and you can do some fun things with the data. For the next three days, I'm going to publish some analysis and insights gleaned from their work. Update: I published an entry about one-hit wonders and pop longevity.

The broad theme of the episode is the information economy, and it's a whirlwind tour of influential people and tech in the pre-Web 1990s. They focus heavily on Microsoft, Apple, and General Magic, with interviews from key players from each. (Plus, great footage of their headquarters and workspaces.) There's plenty of footage of vintage '90s tech in here, including giant cell phones, Windows NT 3.1, the Newton, General Magic's Magic Cap, Corbis, Encarta on CD-ROM, interactive TV and software agents. Some of the highlights, with screenshots, after the video below.

Note: Like all the other materials I post here, this video is completely out-of-print and unavailable commercially, digitized from an old VHS tape. If it ever comes back into print, or the copyright holders contact me, I'll take it down immediately.

The first is ZombieURL, a TinyURL-clone with a twist.

Put in a URL, send it to a friend, and enjoy. Keep a camera nearby to capture the fun. (Here's a Zombie-fied version of Waxy, for example.)

(Warning: We're not responsible for emotional scarring, concussions, or heart failure as a result of using ZombieURL.)

For obvious reasons, I can't share the whole Infocom Drive. But I have to share some of the best parts. It's just too good.

So let's start with the most notorious — Milliways: The Restaurant at the End of the Universe, the unreleased sequel to Infocom's The Hitchhiker's Guide to the Galaxy. For the first time, here's the full story: with never-before-seen design documents, internal emails, and two playable prototypes. Sit back, this might take a while.

Note: I've pieced together this history from emails and notes from the Infocom Drive. I haven't contacted any of the people mentioned, so if you're a primary source or authority, please get in touch so I can make corrections.

Update: Don't miss the comments section. Infocom alumni Dave Lebling, Steve Meretzky, Amy Briggs, and Tim Anderson all comment on the story, Zork co-author Marc Blank helps correct an error, and writer Michael Bywater provides an alternative view of the events.

November 26, 2009:

After much consideration, I've edited this entry to remove the private correspondence that I never sought permission to publish. In my eagerness to share the story, I rushed to post instead of taking the time to reach out to everyone involved, against the recommendations of my source. At the time, I felt that 20 years of distance and the material's historical value was enough to warrant bringing it to light without contacting each individual. I was wrong, and I'd like to apologize to each of the Infocom employees for my lapse in judgment.

On these pages, I've frequently waded into murky legal waters and I'm used to catching a certain amount of heat. But this article cost me far more than I ever anticipated — I lost a very close friend because of it. I deeply regret the error.

-- Andy.

AppDrop is a container for applications developed with the Google App Engine SDK, running entirely on Amazon's EC2 infrastructure. Just like Google's Appspot, anyone can use a modified SDK to deploy their App Engine apps directly to Amazon EC2 instead of Google, and they work without modification.

This proof-of-concept was built in only four days and can be deployed in virtually any Linux/Unix hosting environment, showing that moving applications off Google's servers isn't as hard as everyone thought.

How does it work? Behind the scenes, AppDrop is simply a remote installation of the App Engine SDK, with the user authentication and identification modified to use a local silo instead of Google Accounts. As a result, any application that works with the App Engine SDK should work flawlessly on AppDrop. For example, here's Anderson's Fug This application running on Google App Engine and the identical code running on EC2 at AppDrop.

Of course, this simple portability comes at the cost of scalability. The App Engine SDK doesn't use BigTable for its datastore, instead relying on a simple flat file on a single server. This means issues with performance and no scalabity to speak of, but for apps with limited resource needs, something as simple as AppDrop would work fine.

I spoke to Chris this morning about his project and where he wants it to go. "AppDrop is open-source just like the Google SDK, so I'm hoping someone will come along and take it to the next level," he said. "It wouldn't be hard for a competent hacker to add real database support. It wouldn't be that hard to write a Python adapter to MySQL that would preserve the BigTable API. And while that wouldn't be quite as scalable as BigTable, we've all seen that MySQL can take you pretty far. On top of that, you could add multiple application machines connecting to the central database, and load-balancing, and all that rigamarole."

While this is only a hack, it demonstrates that App Engine developers don't need to live in fear of Google's reprisal. "The upshot is that if you put a lot of time into an App Engine app, and then run afoul of Google, you have alternatives, even if they are more work."

Update: Chris announced the project on his own blog, with some design notes.

April 20, 2008: Addressing concerns with data lock-in, Google announced that large-scale data import and export for will be coming soon to App Engine.

For lack of a better name, let's call them supercuts. (Thanks, Ryan.)

Here are some examples I could find, but I'm sure there must be more. Post 'em in the comments and I'll add them. Bonus points for supercuts with the most clips, the shortest clips, and in additional genres (sports? politics?).

Film

Glengarry Glen Ross - Obscenity Count
Ship, Computer, and Sauce in the first six Star Trek films
Shia LaBeouf in "No No No No"
Requiem for a Dream, montage of every drug montage (meta!)
Big Lebowski, every "fuck"
Casino, every "fuck"
Big Lebowski, Every "dude"
True Romance, every kill from the finale
True Romance, every "fuck"
Rushmore, handjob references (thx, Matt)
Scarface, every "fuck" (thx, oscar)
The Departed, every "fuck" (thx, oscar)
Jay and Silent Bob Strike Back, every "fuck" (thx, MeatFarley)
All the pauses and silence in His Girl Friday from 1940 (thx, progosk)
The Incredibles, buttons, doors, and explosions (thx, Joshua)
Fargo, every "yeah" (thx, Doobybrain)
Charles Bronson Death Wish Bodycount (thx, Dave)
Chris Hefner's Talking Picture (The Road to Ruin), removes all words from a 1938 film (thx, Jamie)
Boondock Saints, every "fuck" (thx, Brandon)
Midnight Run, every "fuck" (thx, Lakawak)
Miscellaneous films, cops turning in their badge and gun (thx, Darrin)
Every "McFly" from the Back to the Future trilogy
Cell phone cliches in horror films

TV

Every "What?" from the LOST series
Legend of Zelda TV show, Excuse Me, Princess!
Knight Rider - Turbo Boosts
CSI: Miami, Caruso's One-Liners
The Simpsons, every couch intro
The Sopranos, Every single whacking
Ojamajo Doremi, every transformation
Star Trek: The Next Generation, Last 10 seconds of every episode of season 1
Every Dragonball Z transformation
Mighty Morphin' Power Rangers, every finishing move from Season 1
Deadwood, every curse in Episode 1, 2, and 3 (thx, Brian W)
LOST, Sawyer says "Son of a Bitch" (thx, Jordan)
Rozen Maiden, every "Desu" (527 times! More context, thx Jason)
Buffy the Vampire Slayer, every mention of "Buffy" from Season 1 (Chuck Jones)
Battlestar Galactica, every "frak" from season 1 (thx, Nowak)
Big Brother's Julie Chen, every "but first" (thx, Cardhouse)
House M.D., every "lupus" reference (thx, engtech)
LOST, Sawyer's nicknames in the first three seasons (thx, oscar)
Rachael Ray, "Mmm!"
MTV's Newport Harbor, 82 "like"s in one episode (thx, Jamie)
Red Dwarf, every "smeg" reference in all 52 episodes (thx, arto)
LOST, Desmond saying "Brother" (thx, cypher)
The Wire, Clay Davis' "Sheeeeeit" (thx, dunk3d)
24, Jack Bauer says "damn it" (thx, jonathan)
The Simpsons - Homer's D'ohs, excerpt from an official episode (thx, Scott)
Scrubs - Every Girls Name to J.D. from Dr. Cox (thx, oscar)
Naruto - Every Rasengan from the filler story arcs (thx, Binkley)
The Hills, without the dialogue (thx, Rex)
Bea Arthur says "Whoa!" on the Golden Girls (thx, BWE)
The Office - "That's What She Said" (thx, Jason)
"I'm not here to make friends" from every reality TV show (thx, Rich)
Dr. Who - "What are you doing here?" (thx, Duncan)
The Sopranos, every profanity
The Royle Family, every "My Arse!" (thx, Dunk)
Stargate SG-1's Teal'c says "Indeed" (thx, Torley)
Kramer's Entrances from Seinfeld (thx, Buzzfeed)
Every "Dude" from Hurley on LOST

Games

Half-Life series, every G-Man sighting (and part 2)
Every Famicon (NES) Game Title Screen

Miscellaneous Sources

Various films, Wilhelm Screams
Various films, The Slow Clap
Various films, NOOO!
Steve Jobs says "Boom!" in Mac keynotes (thx, Jeff)
Clip from Christian Marclay's "Telephones" from 1995 (thx, progosk)
iPhone "Hello" ad, heavily inspired by "Telephones" (thx, progosk)
Phil Collins' "In the Air Tonight," every "Oh Lord" (thx, oscar)
2008 State of the Union, without any speech
Bill Gates says "Uh..." in Rocketboom interview
Collage of "Zoom In and Enhance" scenes

Audio

Only the inhaling from an hour of NPR's All Things Considered (Chuck Jones)
Loveline, every question asked by the hosts (Chuck Jones)
Loveline, every name mentioned in alphabetical order (Chuck Jones)
Loveline, every "Yes" and "No" (Chuck Jones)
Loveline, every number (Chuck Jones)
Loveline, interjections (Chuck Jones)
Loveline, dead air (Chuck Jones)
NWA's Straight Outta Compton, obscenities only (thx, oscar)

Honorable Mention

A commenter points to the work of Chicago artist Chuck Jones, who's created a number of excellent audio and video supercuts he calls Isolation Studies. I've listed them all above.

The "Most Obsessive" award goes to artists Jennifer & Kevin McCoy for their work Every Shot, Every Episode from 2001, a 277 DVD set compiling 10,000 clips from Starsky & Hutch, arranged by categories like "Every Dead Body," "Every Mirror," "Every Gunshot," and "Every Affirmative Response." Other work of theirs includes I Number the Stars, a shot-by-shot index of the first 20 Star Trek episodes in 120 categories, How I Learned, a 10,000 shot inventory of the show Kung Fu in over 100 categories, and Every Anvil, cataloging the violence in 100 Looney Tunes cartoons. Mind-blowing. (Thx, Buzzfeed!)

I've also taken the opportunity to change my URL structure, removing some cruft and giving some additional length to the slugs. All old URLs should redirect, thanks to some mod_rewrite magic and a little PHP.

One change that might affect you is that you'll now be seeing all of my longer articles in the Waxy Links feed, so you might want to unsubscribe from the main feed to avoid getting duplicates. Since I generally only post once a day, and I try to only write things I'd want to link to, this seems like a good alternative to linking to my own posts. If you really don't like this change, please let me know privately and I'll work something out.

What do you think? Nothing's permanent, so I'm all ears.

Update: You're probably seeing some weirdness in the RSS feed. Links appearing in the articles RSS feed, tons of old entries, and links pointing directly to Waxy.org instead of the sites I was linking to. All of these problems are fixed! But feedreaders take time to update, so it'll be resolved as soon as your RSS reader of choice sees the updated feed. Sorry!

In the comments, several people pointed out a noticeable mistake in the video in which the narrator claims Mosaic was developed in 1934. As it turns out, this was fixed in later editions. "I actually came out with a corrected version of that tape," said Ducharme. "In a later version, it was changed to the correct year of 1936. (Kidding, of course, 1993.)"

He also gives some insight into how hard it was to capture video in the early '90s. "We had to rent this gigantic VGA to NTSC converter for about $500 a day to capture those screens," said Ducharme. "It was huge, had to be rolled in and had an operator to work it. Must go now. Am feeling old. Thanks again."

Thanks for the info, Steve! And with that, it's on to Volume 2...

Internet Power Volume 2: Discover the World of Online Education

"Many people believe that the Internet will have the same impact on society as the invention of the printing press did! It will change our world dramatically by making vast amounts of information available to everyone worldwide... if you know how to use it!"

With those breathless words, we start the second volume of Internet Power, the video tutorial series about the Internet from 1995. While the last episode focused on Entertainment, this one teaches you about Education -- museums, libraries, and other research tools.

It's very similar to the first episode with an almost-identical structure, but focusing on a new set of vintage websites. Microsoft's Bret Arsenault appears again, and they focus on Yahoo! again.

Highlights

03:21. Using Yahoo circa 1995 to search for "civil war" sites. The results are in alphabetical order by category name, rather than any attempt at ordering by relevancy.

"We click on the hypertext link, the Civil War Gallery. The URL address for this site is hidden in the link and it automatically takes us to the Civil War Gallery." Updated September 25, 1995!

05:55. "A reminder to you that clicking on links and moving from site to site usually will take a few seconds depending on the speed of your computer's modem. For the purposes of this video, we are moving instantly to each screen with the help of video editing."

06:30. Downloading a 1MB JPG from the Library of Congress and viewing it in ImageView for Windows.

10:15. Listening to a WAV audio greeting from Michael Heyman, the Secretary of the Smithsonian Museum. 40 seconds of filler!

11:35. "This will get you started, but the best thing to do is follow the hypertext links. Explore the Internet and discover your own favorite sites. It's kind of like going to a new city! The streets aren't familiar at first, but after traveling around for a while you'll discover your favorite places in the city that have the activities and interests you're looking for."

12:00. "The Internet has not forgotten about the children of the world." I love this line.

13:00. "This brings us to, of all places, the CIA Website. Don't worry! It's okay that we're here. They keep their top-secret files someplace else."

16:40. "Speaking of science projects, it's pretty difficult going all the way through school without having to do a science project. If this is the case for you, there are ideas on the Internet if you look for them."

20:20. Searching for tuition rates for the University of Washington. In 1994, $969 per quarter for full-time residents, $2,733 for non-residents. (Just for fun, I looked up the current rates. $2,129 for residents, $7,377 for non-residents.)

23:30. Browsing SPRY's Gopher server, explaining how to download JPG files with Gopher.

25:50. "But if you really want to see the power of the Internet for doing research and finding information, just click on the folder 'Libraries of the Internet.' Now hold onto your hats and click the icon that says 'Other Libraries around the US.' Up comes over 100 libraries that you can click on and get even more information of all kinds." He finally concludes, "Never again will you be able to use the excuse of 'Gee, I just couldn't make it to the library last night!'"

27:00. The big ending. "Not bad for 30 minutes, is it? We think you will agree after watching this video that the Internet will change the way we learn forever. Well, we're off to our next website and we hope you enjoy the Internet as much as we do. Remember, in the next century, the person with the most information wins. Good luck on your journey, surf wisely, and have a great time!"

URL Addresses for Websites We Visited

Here's a list of the URLs mentioned in the credits.

Smithsonian, http://www.si.edu
Virtual Science and Math Fair, http://www.educ.wsu.edu/fair_95 (misspelled as www.educ.wsu)
KidsWeb, http://www.npac.syr.edu:80/textbook/kidsweb
Yahoo!, http://www.yahoo.com
Internet College Exchange, http://www.usmall.com/college
University of Washington, http://www.washington.edu
Midlink, http://longword.cs.ucf.edu/~midlink
Web 66, http://web66.coled.umn.edu/schools.html
Welcome to the Planets, http://stardust.jpl.nasa.gov/planets
Library of Congress, http://www.loc.gov (misspelled as www.loc/gov)
Civil War Photo Gallery, http://www.magibox.net/~civilwar/harper.html