This morning, I woke to the news that Archive Team is working to save Upcoming. This is the Internet equivalent of hearing that Marsellus Wallace is sending The Wolf.

For those unfamiliar, Archive Team is a band of rogue archivists and programmers working to rescue dead and dying websites from destruction. To put it mildly, they are very good at what they do.

Led by computer historian/documentary filmmaker Jason Scott, they’ve saved massive sites like GeoCities, Friendster, MobileMe, Fortune City and many others from deletion, and collaborate with the Internet Archive to inject their backups into the Wayback Machine for permanent preservation.

The importance of their work can’t be overstated. While companies like Yahoo work to destroy as much Internet history as possible, Archive Team is the only group actively trying to save it.

To assist their efforts, they’ve developed ArchiveTeam Warrior, a virtual appliance that makes it easy for anyone to help archive dying websites and upload the backups to their server.

Want to help? Install Warrior right now.

It’s dead simple to get up and running, and works on Windows, Mac, and Linux. And because it all runs in a virtual machine, it can’t possibly hurt your system. It will only use your bandwidth and disk space.

After it’s installed, you can choose the “Upcoming” project to start backing up Upcoming.org specifically, or pick “ArchiveTeam’s Choice” to let the team decide. Posterous and Formspring are also dying soon, and that will allow the team to prioritize your work.

I made a little video showing how easy it is to start saving Internet history.

You can track the status of the Upcoming archiving effort in real-time, currently at around 6% of the complete site.

And again, thanks to all the dedicated volunteers of Archive Team for their effort.

Update (April 23): Three days later, the Upcoming archive is complete. Every event, venue, group, and user page is currently being compressed and uploaded in batches to the Internet Archive. Truly amazing.

My next step: to parse the HTML and extract structured data, distributed that database, and build something off it to make the community-contributed material accessible after Yahoo shuts it down.

So, Yahoo’s finally decided to close Upcoming.org, the events community I started nearly ten years ago. And, in Yahoo’s typical fuck-off-and-die style, they’re doing it with 11 days notice, no on-site announcement, and no way to back up past events.

I knew its closure was inevitable after the infamous sunset slide, but never knew when it would happen. Like a newspaper prepping for a sick celebrity, this obituary’s been sitting in my drafts folder for months, waiting for its sad publication day.

The last five years were hard on Upcoming. After Gordon Luk, Leonard Lin, and I left at the end of 2007, the site quickly started to fall apart. The social features that made Upcoming unique were minimized, or removed entirely, by a series of redesigns. Spam, like creeping kudzu, was left unchecked and spread across the site. Fortunately, the final catastrophic redesign never made its way out of beta.

By 2009, the only people using Upcoming were event promoters and spammers. (Especially depressing considering self-promotion was banned entirely for its first two years.)

Frustratingly, nothing’s come to take its place. Potential competitors like Plancast and Going closed their doors, while others never grew an organic community. Some sites carved off a piece of Upcoming: Facebook’s private events, Songkick’s concerts, and Lanyrd’s fantastic conference coverage.

But, for me, finding events I care about feels like 2002 again. I’m missing geeky events I’d love, and when I travel to a new city, I’m back to digging through the calendar listings of my local weekly newspapers. It blows my mind that the problem Upcoming solved — surfacing interesting events in a city, driven by public social activity — is an unsolved problem again.

And now, Yahoo will quietly take Upcoming off life support, an opportunity squandered.

Bleeding Purple

It’s hard to believe now, but there was a time when Yahoo was actually pretty cool, in its own dorky Silicon Valley way.

By 2005, when we started talking to Yahoo, they’d made a series of thoughtful hires, including PHP creator Rasmus Lerdorf, Jeremy Zawodny, Tom Coates, Simon Willison, and future Etsy CEO Chad Dickerson. Cameron Marlow, Jeffery Bennett, and Mor Namaan were doing pioneering work at Yahoo Research. They acquired Flickr, bringing some of the most talented and creative people in technology to help change the company from the inside, including Cal Henderson, Heather Champ, and founders Caterina Fake and Stewart Butterfield. A month after we came in, they acquired Del.icio.us.

Clueful people were making their way up to the executive level, too. Future Bandcamp founder Ethan Diamond led the redesign of Yahoo Mail, future Topspin CEO Ian Rogers was managing Yahoo Music, and and Bradley Horowitz, now VP of Product at Google, was taking over big pieces of the company. Yahoo was an exciting place to be.

Upcoming was a side project, created during my day job at a financial company. After my son was born, I had no time to work on Upcoming at all, even as the community grew. Spammers started to discover the site, as bug reports and support requests piled up unanswered. The opportunity to work on my own project full-time was a dream come true.

And Yahoo seemed like a perfect home for Upcoming — they’d promised resources to grow the community, we’d get to work at a promising tech giant with some of our favorite people, and the acquisition price was small but seemed fair. Coming into Yahoo, we were hopeful.

It wasn’t clear how dysfunctional the rest of Yahoo was until we’d settled in, and there was no indication how horrible they’d soon become in the years to follow. This was long before they gave up dissidents to the Chinese government, closed Geocities, weaponized their patents, “sunsetted” Delicious, and a number of other awful decisions.

In hindsight, selling Upcoming to Yahoo was a horrible mistake. Selling your company always means sacrificing control and risking its fate, and as we now know, online communities almost always fail after acquisition. (YouTube is the rare exception, albeit one with billion-dollar momentum.) But Yahoo was a particularly horrible steward for the community.

I built Upcoming because it scratched a personal itch, and I was delighted when so many others found it useful. For the small group of old-schoolers that remember it in its prime, Upcoming made their lives better. I’ve heard stories of people finding friends and spouses through Upcoming, people lonely in a new city tapping into new communities, impromptu parties gaining momentum.

I’m going to miss it.

Archiving Upcoming

Upcoming stopped being relevant long ago, and part of me is happy that Yahoo’s putting this bastardized version of the site out of its misery. (In case your memory’s foggy, compare how it looked when we left to its current state.)

What really upsets me is that the archived events will soon be taken offline, and with no way to back it up. Ten years of history will be gone in 11 days. Good URLs never die, and I’m frustrated that every link to Upcoming will soon 404.

I’ve reached out to Yahoo multiple times over the last few months about re-acquiring the Upcoming.org domain and event database, but they were less than receptive.

I would love to create a permanent archive of Upcoming, with a clean responsive layout and some month-by-month analysis and visualization of the site’s history, but getting the metadata’s proving much more difficult than I thought.

All of Upcoming’s events and venues use autoincremented ids, making it dead simple to generate a list of URLs to scrape. But Yahoo’s security makes scraping a challenge. Every time I’ve tried to back up pages, I can only grab a few files with curl or httrack before Yahoo starts serving blank responses.

Note that scraping the HTML alone won’t provide the full list of attendees for popular events, which are displayed via Javascript. For example, to get all the metadata for this event, you’d need to scrape both the event page for the event details and this XML for the attendees.

If you have any idea how to scrape Upcoming’s events, or can get me a dump in any form, please get in touch ASAP. Anonymity guaranteed.

Update: Archive Team is working to save Upcoming, and they need your help in the rescue efforts.

Last month, I spoke at Creative Mornings/Portland about copyright, fair use, and remix culture. It started as a riff on my No Copyright Intended post, and ended up something much bigger. I really like how it came out, I hope you do too.

Watch it full-screen HD on YouTube.