Last Friday, a YouTube user named eeplox posted a question to the support forums, regarding a copyright complaint on one of his videos. YouTube’s automated Content ID system flagged a video of him foraging a salad in a field, claiming the background music matched a composition licensed by Rumblefish, a music licensing firm in Portland, Oregon.

The only problem? There is no music in the video; only bird calls and other sounds of nature.

Naturally, he filed a dispute, explaining that the audio couldn’t possibly be copyrighted.

The next day, amazingly, his claim was rejected. Not by YouTube itself — it’s unlikely that a Google employee ever saw the claim — but from a representative at Rumblefish, who reviewed the dispute and reported back to YouTube that their impossible copyright for nonexistent music was indeed violated.

Back at YouTube, eeplox found himself at a dead end. YouTube now stated, “All content owners have reviewed your video and confirmed their claims to some or all of its content.” No further disputes were possible, the case was closed.

Whether caused by a mistake or malice, Rumblefish was granted full control over eeplox’s video. They could choose to run ads on the video, mute the audio, or remove it entirely from the web.

A History of Screw-Ups

On Sunday night, Reddit took notice. Within hours, the thread was on the homepage, commenters were freaking out and, to his credit, Rumblefish CEO Paul Anthony was fielding questions in an IAmA interview until 2:30am.

His argument: One of Rumblefish’s Content ID reps made a mistake by denying the dispute, and they released the claim on Sunday night. “We review a substantial amount of claims every day and the number is increasing significantly,” said Anthony. “We have millions of videos now using our songs as soundtracks and keeping up is getting harder and harder.”

This is the latest in a long series of foibles or outright abuses of YouTube’s Content ID system. Content ID was intended to help copyright holders manage the chaos of YouTube. They’d provide copies of their audio and video for analysis, which would then algorithmically match newly-uploaded videos. If a match was found, rightsholders could automatically block the video or, increasingly, claim money from video advertising.

Content ID’s monetization was a huge boon for copyright holders. Uploaders could keep their videos online, while copyright holders profited from the creative reuse of their work.

But the last couple years have seen a dramatic rise in Content ID abuse, using it for purposes that it was never intended. Scammers are using Content ID to steal ad revenue from YouTube video creators en masse, with some companies claiming content they don’t own, deliberately or not. The inability to understand context and parody regularly leads to “fair use” videos getting blocked, muted or monetized.

Bypassing the DMCA

The problem is that media companies and scammers are using Content ID as an end run around the DMCA.

With the DMCA, the process works like this. A rightsholder could file a claim against a video with YouTube, and YouTube would immediately take the video offline. If there was a mistake, the uploader could file a counter-notice. The video would then be restored by YouTube within 10-14 business days of the counter-notice, unless it went to court.

It wasn’t perfect, by any means, but it was fair. Disputes could always be appealed, and both parties were given equal power. And if a claimant lied about owning the copyright to the material in question, they could face perjury charges.

The current system, led by Content ID, tips the balance far in favor of the claimant.

Rumblefish never needed to prove they were the copyright holder, but were still given ultimate control over the video’s fate. Uploaders can dispute claims, but the only people reviewing claims are the Content ID partners that filed the claim in the first place, who are free to deny them wholesale.

A Simple Fix

The solution is simple: if a copyright holder wants to pursue a disputed Content ID match, they should file a DMCA claim. That’s the only way to guarantee their rights, and make the copyright holder legally responsible for telling the truth.

In fact, this is exactly how YouTube says that Content ID “fair use” claims should work. In practice, this doesn’t appear to be true any longer. Content ID partners, of course, can file a DMCA notice at any time, but why bother if they can reject the counter-claims themselves?

(Preferred partners like Universal Music Group can go a step further and block videos directly without filing a claim.)

This problem has been on YouTube’s radar for at least two years, but it’s only getting worse as unsavory companies discover this nascent business model. Claim copyright on media you may or may not own, and let Content ID do the rest.

By letting Content ID partners have the final word, and not trusting their own users, YouTube is violating its trust with its community and damaging fair use in the process.

Update

I originally published this article over at Wired, where a commenter pointed out that this process may actually violate YouTube’s “safe harbor” granted through the DMCA. If they choose to ignore disputes, they’re effectively giving content providers an end run around fair use and the DMCA.

Selfish Crab wrote:

It seems like by providing the Content ID system, Youtube was trying to pre-emptively identify copyrighted material, like a first-pass dispute system. Their lawyers probably concluded that so long as the content ID system falls back onto DMCA takedown procedure, they are still in compliance with the DMCA sufficiently to retain their safe harbor.

So if Content ID claim disputes do not fall back onto DMCA takedown, as Andy’s article suggests, there’s a case to be made that YouTube no longer has liability protection from users. It is a whole another can of worms to analyze what a legal claim against youtube would look like. You’d have to look at the YouTube Terms of Service (i.e., the contract) to see if maybe they contracted around this problem already, you’d have to figure out damages, etc etc. Or I guess you can just raise a shitstorm and that’s enough of a moral victory.

In a Google+ comment last December, senior copyright counsel for Google and former EFF staff attorney Fred von Lohmann acknowledged the problem.

Yes, we’re aware of that problem in the Content ID dispute process and are looking at what we can do to fix it. It’s the result of a complicated collision of how to handle geographically limited Content ID claims, disputes, and global DMCA removals. Turns out to be a hard problem to figure out. But we’re thinking on it.

Virginia law student Patrick McKay got in touch with Annie Baxter, a public relations manager at YouTube, about this issue.

This is one of those corner-case outcomes that emerges from several different rules, none of which was intended to yield the result you’ve encountered (i.e., DMCA takedowns are global, but Content ID ownership claims are territorial). Unfortunately, addressing it YouTube-wide is going to take some time, both for pondering and implementing.

So while we can promise you that we’re thinking about this, we can’t promise you a fix or time-table. And feel free to tell the OVC we’re looking at it and trying to come up with something.

In the meantime, anyone in the Content ID program is offered free rein to claim copyright on your videos and profit directly from them. I’m hoping this gets cleared up soon.

So, I made a weird new thing with my 15-year-old nephew, Cooper McHatton. It’s experimental and has lots of rough edges, but quite frankly, I’m tired of working on it, so here you go.

Playfic is a community for writing, sharing, and playing interactive fiction games (aka “text adventures”) entirely from your browser, using a “natural language”-inspired language called Inform 7.

Inform 7 is incredibly awesome and weird. For example, this is a fully functional game:

East of the Garden is the Gazebo. Above is the Treehouse. A billiards table is in the Gazebo. On it is a trophy cup. A starting pistol is in the cup. In the Treehouse is a container called a cardboard box.

Type that into Playfic, and you end up with this simple game, ready to send to the world.

The official documentation is extensive, with a great manual and recipe book. I’ve collected a list of resources to help you get started.

For now, there’s very little documentation on Playfic itself, but you can click the “View game source” link on every game to see how it was made, and Cooper’s adding sample games from the official Recipe Book.

My hope is that Playfic opens up the world of interactive fiction to a much wider audience — young writers, fanfic authors, and culture remixers of all ages.

While the language can be tricky, building simple games is surprisingly easy. Cooper had never coded anything or made a game before trying Playfic, and within 30 minutes of futzing around, he’d made his first game.

Some stuff is broken and missing, but I’d love to hear what you make of it. Open to any and all feedback. Go make some games!

The other day, I tried out Unroll.me, a clever new service that reads your inbox to let you unsubscribe from mailing lists and other unwanted e-mail flotsam with a single click.

As I was about to connect my Gmail account, my finger hovered over the “Grant access” button.

Wait a second. Who am I giving access to my Gmail account, anyway? There was no identifying information on their site — no company address, no team page listing the names of its team members, and broken links to their privacy policy or terms of service.

For all I knew, it could be run by unscrupulous spammers or an Anonymous troll looking for lulz. And I was about to give them unfettered access to eight years of my e-mail history and, with password resets, the ability to access any of my online accounts?

I had to dig around online to find out who’s behind it, and fortunately, Unroll.me is a totally legit NYC-based startup providing a useful service. I spoke to Perri Blake Gorman, Unroll.me’s cofounder and CMO, who assured me they’ll add all the company information as they roll out their public beta.

But since Gmail added OAuth support in March 2010, an increasing number of startups are asking for a perpetual, silent window into your inbox.

I’m concerned OAuth, while hugely convenient for both developers and users, may be paving the way for an inevitable privacy meltdown.

The Road to OAuth

For most of the last decade, alpha geeks railed against “the password anti-pattern,” the common practice for web apps to prompt for your password to a third-party, usually to scrape your e-mail address book to find friends on a social network. It was insecure and dangerous, effectively training users how to be phished.

The solution was OAuth, an open standard that lets you grant permission for one service to connect to another without ever exposing your username or password. Instead of passwords getting passed around, services are issued a token they can use to connect on your behalf.

If you’ve ever granted permission for a service to use your Twitter, Facebook, or Google account, you’ve used OAuth.

This was a radical improvement. It’s easier for users, taking a couple of clicks to authorize accounts, and passwords are never sent insecurely or stored by services who shouldn’t have them. And developers never have to worry about storing or transmitting private passwords.

But this convenience creates a new risk. It’s training people not to care.

It’s so simple and pervasive that even savvy users have no issue letting dozens of new services access their various accounts.

I’m as guilty as anyone, with 49 apps connected to my Google account, 80 to Twitter, and over 120 connected to Facebook. Others are more extreme. My friend Sam is a developer at Kickstarter, and he authorized 148 apps to use his Twitter account. Anil counted 88 apps using his Google account, with nine granted access to Gmail.

For Twitter, the consequences are unlikely to be serious since almost all activity is public. For Facebook, a mass leak of private Facebook photos could certainly be embarrassing.

But for Gmail, I’m very concerned that it opens a major security flaw that’s begging to be exploited.

The Privacy Danger

A long list of services, large and small, request indefinite access to your Gmail account.

I asked on Twitter and Google+ for people to check their Google app permissions to see who they’ve granted Gmail access to. The list includes a range of inbox organizers, backup services, email utilities, and productivity apps: TripIt, Greplin, Rapportive, Xobni, Gist, OtherInbox, Unsubscribe, Backupify, Blippy, Threadsy, Nuevasync, How’s My Email, ToutApp, ifttt, Email Game, Boomerang, Kwaga, Mozilla F1, 0boxer, Taskforce, and Cloudmagic.

Once granted, all of these services are issued a token that gives unlimited access to your complete Gmail history. And that’s where the danger lies.

You may trust Google to keep your email safe, but do you trust a three-month-old Y Combinator-funded startup created by three college kids? Or a side project from an engineer working in his 20 percent time? How about a disgruntled or curious employee of one of these third-party services?

Any of these services becomes the weakest link to access the e-mail for thousands of users. If one’s hacked or the list of tokens leaked, everyone who ever used that service risks exposing his complete Gmail archive.

The scariest thing? If the third-party service doesn’t discover the hack or chooses not to invalidate its tokens, you may never know you’re exposed.

In the past, Gmail’s issued security warnings to accounts being accessed from multiple IP addresses. I spoke to OtherInbox founder Joshua Baer, and he said that Google’s eased up on the warnings because of the prevalence of third-party services.

It’s entirely possible for someone with a stolen token to read, search, and download all your mail to their server for months, and you’d never find out unless they exposed themselves, or you were diligently auditing your “Last account activity” history.

Stay Safe

Clearly, we’re not going to stop using awesome new utilities just because there’s a privacy risk. But there are best practices you can follow to stay safe.

• Clean up your app permissions. The best thing you could do, right now, is to log into each service you care about and revoke access to the apps you no longer use or care about, especially those that have access to Gmail. Finding the permissions pages can be tricky, but the nice folks at MyPermissions.org made a handy dashboard linking to every one.
• Think before you authorize. Before authorizing an account, find out who you’re granting access to. Look for a staff page, contact address, and take a look at the privacy policy to make sure they’re not sharing or selling your info with third parties. Bonus points if they outline their security policies and offer a way to disconnect service from within the app. If anything seems off, don’t do it.
• When in doubt, change your password. Have a feeling that someone might be reading your mail, but not sure which app is to blame? Changing your password instantly invalidates all your Google and Facebook OAuth tokens, though Twitter tokens persist after password changes.

Google could improve, as well. Their permissions page is too hard to find, even for experienced users, and it’s impossible to see which apps have accessed your account recently.

Facebook does an excellent job with this, but Google only shows you the IP address and the protocol it used to connect. Surfacing this information, as a periodic e-mail or on-site notification, would go a long way to averting a potential disaster.

The Greatest Troll of All

So, I originally published everything above over on my Wired column yesterday, but I left off something else I’ve been thinking about.

While I think a compromised database is the most likely scenario, there’s another possibility that disturbs me more.

Imagine that a brand new service pops up, offering a simple, fun service that uses your Gmail account. Maybe a neat visualization like Tout’s Year in Review, or maybe something more practical like sending all your attachments to Dropbox.

But it’s all just a giant troll, where the app’s creators are silently running targeted searches, downloading your mail, and looking for compromising photos and sensitive documents behind-the-scenes. They could collect the documents for months or years, and then release it all online in an anonymous blast. Lulz!

You’d likely never find out where the data came from, and the perpetrators would never be caught. Hell, if you’ve Gmail-authed a questionable app, this could be happening to you right now and you’d never know. Whee!

Every year, the MPAA tries desperately to stop Oscar screeners — the review copies sent to Academy voters — from leaking online. And every year, teenage boys battling for street cred always seem to defeat whatever obstacles Hollywood throws at them.

For the last 10 years, I’ve tracked the online distribution of Oscar-nominated films, going back to 2003. Using a number of sources (see below for methodology), I’ve compiled a massive spreadsheet, now updated to include 310 films.

This year, for the first time, I’m calling it: after three years of declines, the MPAA seems to be winning the battle to stop screener leaks. But why?

A record 37 films were nominated this year, and the studios sent out screeners for all but four of them. But, so far, only eight of those 33 screeners have leaked online, a record low that continues the downward trend from last year.

(Disclaimer: Any of this could change before the Oscar ceremony, and I’ll keep the data updated until then.)

They may be winning the battle, but they’ve lost the war.

While screeners declined in popularity, 34 of the nominated films (92 percent) were leaked online by nomination day, with 25 of them available as high-quality DVD or Blu-ray rips. Only three films — Extremely Loud & Incredibly Close, My Week with Marilyn and W.E. — haven’t leaked online in any form (yet!).

If the goal of blocking leaks is to keep the films off the internet, then the MPAA still has a long way to go.

There are a number of theories about what’s causing the decline.

It could be attributed to tighter controls — personalized watermarks, the aggressive prosecution of leakers, and greater awareness of the risks for Academy voters.

But the MPAA may have little to do with the decline. Oscar-nominated films could be coming out earlier in the year, making screeners less important.

Or maybe the interests between the mainstream downloader and industry favorites is diverging? If the Oscars are mostly arthouse fare and critical darlings, but with low gross receipts, they’ll be less desirable to leak online. It would be very interesting to track the historical box office performance of nominees to see how it affects downloading. (Maybe next year!)

The continuously shrinking window between theatrical and retail releases may be to blame. After all, once the retail Blu-ray or DVD is released, there’s no reason for pirate groups to release a lower-quality watermarked screener.

The chart below tracks the window between U.S. release and its first DVD/Blu-Ray leak online, which shows how the window between theatrical and retail release dates is slowly closing since 2003.

Whatever the reason, online movie releasing groups are taking longer to pirate movies than ever. When I first started tracking releases in the early- to mid-2000s, the median time between theatrical release to its first leak online was 1 to 2 days. Now, that number’s crept up to over three weeks.

The rise in leak time correlates with a dip in popularity for lower-quality sources, like camcorder-sourced footage. This year, only eight of the 37 nominees (21 percent) were sourced from camcorder footage. (This is likely because there are fewer blockbuster nominees than in the mid-2000s.)

As the industry slowly transitions from physical media to streaming video, it’ll be interesting to see if the downward trend continues, or if the ease of capturing streaming video spawns a new renaissance for screeners. Last year, Fox Searchlight distributed screeners with iTunes, and all were quickly and easily pirated.

The Data Dump

Skeptical of my results? Want to dig into it yourself? Good! Here’s the complete dataset, available on Google Spreadsheets or downloadable as an Excel spreadsheet or comma-separated text file.

Methodology

I include the full-length feature films in every category except documentary and foreign films (even music, makeup, and costume design).

I use Yahoo! Movies for the release dates, always using the first available U.S. date, even if it was a limited release, falling back to the first available U.S. date in IMDB.

All the cam, telesync, and screener leak dates are taken from VCD Quality, supplemented by dates in ORLYDB. I always use the first leak date, excluding unviewable or incomplete nuked releases.

The official screener release dates are from Academy member Ken Rudolph, who kindly lists the dates he receives each screener on his personal homepage. Thanks again, Ken!

For previous years, see 2004, 2005, 2007, 2008 (part 1 and part 2), 2009, 2010, and 2011.

Today, you’re going to hear a million solid reasons why SOPA and PIPA — the two proposed bills sponsored by the entertainment industry to censor the web — have to die. Wikipedia, Google, Reddit, craigslist, Metafilter, and many, many more have made their cases. Here’s mine.

Virtually every project I’ve ever worked on is threatened by this legislation:

Upcoming.org faced copyright complaints for event posters and listings that users added to the site.

Kickstarter gets DMCA takedowns from artists who find their work used in pitch videos, and from project founders quarreling with each other.

Supercut.org indexes hundreds of video remixes that reuse copyrighted content.

Kind of Bloop faced a lawsuit over the cover art.

And here on Waxy.org, I’ve had a number of battles over copyright. Among them, I received a cease-and-desist from EMI for being the first person to host DJ Danger Mouse’s Grey Album on the web, from Disney for hosting the Kleptones’ Night at the Hip-Hopera, and from Bill Cosby for hosting House of Cosbys, which was clearly fair use as a parody.

Every cease-and-desist and DMCA request I’ve received wasn’t fun to get in my inbox, but it allowed me to deal with the issues directly with the copyright holder or using the due process of the court system.

Imagine, instead, a world where a bill like SOPA or PIPA passes. A copyright holder could bypass due process entirely, demanding that search engines stop linking to my sites, ad providers drop me, and force DNS providers not to resolve my domain name. All in the name of stopping piracy.

The chilling effect would be huge.

Every online community that allows for community-contributed content — discussion forums, imageboards, Usenet newsgroups, photo sharing communities, video sites, and many more — would be forced to pre-emptively self-censor, shut down, or risk getting blown off the net entirely.

That fucking sucks.

Everything I love about the web requires the unfettered freedom to build new ways to let people express themselves, and with that, comes the risk of copyright infringement.

Breaking the web isn’t a solution.

Please take 10 minutes today to call your representatives — or show up in person! –and let them know you won’t stand for this. SOPA and PIPA must die.

Streaming music services like Spotify and Rdio are transforming the way we listen to music, but spotting differences in their catalogs is nearly impossible for the casual listener. The licensing landscape is constantly shifting, with songs appearing and disappearing as labels try to make up their minds.

To help you decide which service is right for you, I’m using the developer APIs provided by each service to go crate-digging into each catalog to see which service comes out on top.

Last time, we looked at 5,000 critically loved albums on both services, with Rdio barely edging ahead of Spotify. That’s great for music geeks who can’t live without “Marquee Moon,” “Bitches Brew” and “In the Aeroplane Over the Sea.” But it leaves more mainstream, single-oriented music fans out in the cold.

If you love pop music, this is your week. We’re digging into 56 years of Billboard charts, searching Spotify and Rdio for every year’s top 100 from 1955 to 2011 — from Elvis Presley and Dean Martin to Rick Ross and Waka Flocka Flame.

How It Works

The Billboard chart data comes from the Whitburn Project, a group of obsessive music collectors who have been quietly compiling historical chart data on Usenet since 1998. Originally intended to help complete their MP3 collections, they used multiple sources to create a spreadsheet of over over 38,500 songs dating back to 1890, with 112 columns of raw data, including each song’s duration, beats per minute, songwriters, label, and week-by-week chart position.

Here’s a sample of the most recent Whitburn spreadsheet from November 11, 2011, so you can see the fields they entered.

With the spreadsheet, I selected the top 100 songs that stayed at the top of the charts the longest each year starting in 1955, and pulled it into a database for easy manipulation.

With these 5,700 songs, I then wrote a script to search the Rdio and Spotify APIs for each track. To standardize artist and song names, I used the Echo Nest’s Song.search API. As before, I’m only checking U.S. availability, since Rdio is limited to the United States and Canada only.

Disclaimer: Variations in artist and song names can lead to some missed results, and false positives can crop up due to karaoke versions and tribute bands. I’ve tried to weed out most of the bad results, but didn’t check all 5,700 results by hand. That said, it doesn’t seem like any error favors Spotify or Rdio, so the results should be fair, if imperfect.

Results

Of the 5,700 songs in the top 100, 5,026 (88 percent) were available on both Spotify and Rdio. An additional 81 (1.4 percent) were only on Spotify, and 100 (1.7 percent) only available on Rdio. If we limit it to only the 570 top-10 singles, 518 songs (over 90 percent) were available on both Spotify and Rdio.

The chart below shows the percentage of the top 100 available per year on Spotify and Rdio. At a glance, you can see how deep both of their catalogs are. It’s very rare for either service to have less than 80 percent of the top 100 in a given year. (Note that the Beatles singlehandedly lower their coverage in the mid- to late-1960s.)

Here’s the average percentage by decade:

Let’s start by looking at the holdouts, the top-charting artists that aren’t available for streaming on either service. As in the album analysis, The Beatles top the list with 35 missing hits, but the rest of the list is very different. All 11 of the Eagles’ top hits are unavailable, Bob Seger fans will be bummed to hear his 10 (!) charting singles are missing, and most of the Red Hot Chili Peppers’ post-1991 hits are unavailable for streaming. The Dave Clark Five’s eight hits from the mid-1960s are all missing, and Aaliyah’s estate is apparently protective of her work, blocking access to her eight big singles.

Other surprising holdouts: Hootie and the Blowfish, Joan Jett, and Roberta Flack. A handful of one-hit wonders are missing entirely, depriving the world of songs like Another Bad Creation’s 1990 debut “Iesha” and Rick Dees’ “Disco Duck” from 1976.

The Exclusives

Both services stream virtually every song every to appear on the Billboard charts, but they don’t overlap perfectly. Each have secured different licenses with record labels, giving each exclusive access to some songs and artists.

If you want to hear the 14 singles released by Paul McCartney, solo and with Wings, you can only hear them on Rdio. Same for LeAnn Rimes, Monica, and Fergie. Spotify, on the other hand, didn’t have exclusive access for any artist with more than two charting singles in the yearly top 100 charts.

Below, I’ve listed the top 20 tracks exclusive to each service, ordered by their overall yearly ranking.

Only on Rdio

Only on Spotify

Paul McCartney — My Love (#3, 1973) Paul McCartney — Say Say Say (#4, 1983) Monica — The First Night (#4, 1998) Christina Aguilera — Lady Marmalade (#6, 2001) Kyu Sakamoto — Sukiyaki (#7, 1963) Monica — Angel Of Mine (#7, 1999) Fergie — London Bridge (#7, 2006) *NSYNC — It’s Gonna Be Me (#11, 2000) Paul McCartney — Coming Up (Live At Glasgow) (#12, 1980) LeAnn Rimes — How Do I Live (#12, 1997) Fergie — Big Girls Don’t Cry (#12, 2007) Wings — With A Little Luck (#13, 1978) Divine — Lately (#13, 1998) Red Hot Chili Peppers — Under The Bridge (#20, 1992) LL Cool J — Loungin’ (#20, 1996) Monica — For You I Will (#20, 1997) Enrique Iglesias — Hero (#21, 2001) Paul McCartney — Band On The Run (#22, 1974) Merril Bainbridge — Mouth (#23, 1996) Wings — Listen To What The Man Said (#24, 1975)

Mariah Carey — Don’t Forget About Us (#7, 2005) Steve Miller Band, The — Abracadabra (#9, 1982) Patti Austin — Baby, Come To Me (#10, 1983) Dr. Dre — Nuthin’ But A G Thang (#14, 1993) Shocking Blue, The — Venus (#20, 1970) Mike & The Mechanics — The Living Years (#24, 1989) Salt ‘N Pepa — Shoop (#29, 1993) Ashlee Simpson — Pieces Of Me (#33, 2004) String-A-Longs, The — Wheels (#36, 1961) Irene Cara — Fame (#38, 1980) Climax Blues Band — Couldn’t Get It Right (#42, 1977) Yael Naim — New Soul (#43, 2008) Madonna — Don’t Cry For Me Argentina (#48, 1997) Dr. Dre — Dre Day (#49, 1993) Technotronic — Move This (#50, 1992) Erykah Badu — Love Of My Life (An Ode To Hip Hop) (#54, 2003) Paperboy — Ditty (#55, 1993) Johnny Thunder — Loop De Loop (#57, 1963) Tee Set, The — Ma Belle Amie (#59, 1970) Gerry and the Pacemakers — Ferry Across the Mersey (#61, 1965)

Conclusion

Both services do an extraordinary job at including music history’s most popular songs. Virtually every song was available on Spotify and Rdio, a huge change from the previous album-oriented analysis. Again, much to my surprise, Rdio comes out slightly on top. Spotify’s international catalog fills most of these gaps, so expect things to heat up rapidly over the next year as they secure more of those licenses for the United States.

Have any questions about this analysis, or anything missing you’d like to see? Leave a comment and let me know.

(Note: This was originally published for my column at Wired.)

The new generation of streaming music services like Spotify, Rdio, and MOG have more music than you could consume in a lifetime. But how much of it would you really want to listen to?

There’s no shortage of great roundups and reviews showing the pros and cons of each service, but they rarely talk specifically about the different music you can find on each. They’ve all built impressive catalogs, but it’s nearly impossible to tell from casual browsing which artists and albums are exclusives for each.

Fortunately, both Rdio and Spotify offer powerful developer APIs, making it simple to compare the two. (Sadly, MOG doesn’t offer an API, so isn’t included.)

For this test, I needed a large set of popular, well-loved albums to test. I used the top 5,000 albums from Rate Your Music, the quirky 11-year-old online community dedicated to rating and reviewing music. These albums span all genres, from klezmer to chiptune, with a total of 2,282 different artists across 70 years of recorded music.

I used the Spotify and Rdio search APIs to look up each album, and checked their streaming availability in the United States. (Rdio uses the IP address to determine country of origin, making it impossible to query other countries. Spotify, on the other hand, returns a list of every region the album’s available.)

Note: The results aren’t perfect. Spotify and Rdio often have slight differences between artist and album names, which can deliver false positives. Let me know if you spot anything amiss and I’ll correct it.

Results

Of the top 5,000, about 44% were available on both Spotify and Rdio. 4.8% of the albums were only available on Spotify, while a further 6.8% were only available on Rdio. Overall, 56% of the albums were streamable on at least one of the services.

Labels are still withholding most or all of the albums from many popular artists. The Beatles, King Crimson, AC/DC, The Eagles, Tool, De La Soul, Peter Gabriel, Led Zeppelin, and Metallica are nowhere to be found, as well as most of the best albums by The Kinks. Music geeks will be sad to discover that Frank Zappa, Coil, Spacemen 3, and Joanna Newsom are all missing, as well. This landscape will constantly shift as labels change their minds; Arcade Fire was added to Spotify yesterday, and more than 200 indie labels left the streaming services last month.

But what about albums that are exclusive only to one service? The results surprised me. Spotify has a reputation for having a deeper catalog, but at least for historic critically-regarded albums, Rdio has a better selection of both popular and obscure artists. More albums in the top 5,000 were available on Rdio, and they offer exclusive access in the U.S. to huge acts like Bob Dylan, Pink Floyd, the White Stripes, and Queen.

Top Exclusive Artists

Here’s a list of the top 20 artists exclusive to each service, with the number of exclusive albums in parentheses.

Only on Rdio	Only on Spotify
Bob Dylan (12) Pink Floyd (8) Bruce Springsteen (7) Miles Davis (6) The Gathering (5) Blind Guardian (4) Can (4) William Basinski (4) Iced Earth (4) Stars of the Lid (3) The White Stripes (3) John Williams (3) Queen (3) Nevermore (3) Thelonious Monk (3) Charles Mingus (3) Bill Hicks (3) John Coltrane (2) Camel (2) Keith Jarrett (2)	My Dying Bride (4) Miles Davis (4) Candlemass (3) Funkadelic (3) The Pretty Things (3) Current 93 (3) Darkthrone (3) Underworld (3) Katatonia (3) CunninLynguists (3) Charles Mingus (2) Mahavishnu Orchestra (2) The Jesus Lizard (2) The Misfits (2) Klaus Schulze (2) John Coltrane (2) Galaxie 500 (2) Silvio Rodríguez (2) Secos & Molhados (2) maudlin of the Well (2)

Note that artists like Miles Davis and John Coltrane appear on both lists because of how prolific they were. Both are well-represented in Spotify and Rdio, but some critically-adored out-of-print albums are unavailable on both.

Top Exclusive Albums

Digging into the albums, Rdio wins again. Nine of the top 100 albums are only found on Rdio, while only one is exclusive to Spotify. In fact, there are only 32 albums in the top 1,000 available on Spotify alone. Below is the top 30 for each service, along with their Rate Your Music ranking.

Only on Rdio

Only on Spotify

Pink Floyd – The Dark Side of the Moon Pink Floyd – Wish You Were Here Bob Dylan – Highway 61 Revisited Bob Dylan – Blonde on Blonde The Clash – London Calling Bob Dylan – Bringing It All Back Home Bob Dylan – Blood on the Tracks Pink Floyd – Animals Bob Dylan – The Freewheelin’ Bob Dylan Bob Dylan – Another Side of Bob Dylan Bob Dylan – The Times They Are A-Changin’ Dr. Dre – The Chronic Stars of the Lid – The Tired Sounds Of Camel – Moonmadness The White Stripes – Elephant Bon Iver – For Emma, Forever Ago John Williams – Raiders of the Lost Ark Popol Vuh – Hosianna Mantra Jethro Tull – Nothing Is Easy: Live at the Is… Albert King – Born Under a Bad Sign Keith Jarrett – Vienna Concert Dead Kennedys – Plastic Surgery Disasters Thin Lizzy – Black Rose: A Rock Legend Magic Sam – West Side Soul Bob Dylan & The Band – The Basement Tapes Eric Dolphy – Out There Blind Guardian – Live Devin Townsend – Terria Strapping Young Lad – City Pretenders – Pretenders

The Zombies – Odessey and Oracle Candlemass – Nightfall Funkadelic – Standing on the Verge of Getting… The Jesus Lizard – Goat The Pretty Things – Parachute The Jazz Composer’s Orchestra – The Jazz Comp… Klaus Schulze – X Sodom – Agent Orange Danny Elfman – Edward Scissorhands Galaxie 500 – Today Current 93 – All the Pretty Little Horses Secos & Molhados – Secos & Molhados maudlin of the Well – Bath Sun Kil Moon – Ghosts of the Great Highway Anathema – Alternative 4 Darkthrone – A Blaze in the Northern Sky The Byrds – Fifth DimensionMost Popular The Gun Club – Miami Autopsy – Severed Survival My Dying Bride – Turn Loose the Swans The Jesus Lizard – Liar Vektor – Black Future maudlin of the Well – Leaving Your Body Map Jean Michel Jarre – Oxygene 16 Horsepower – Secret South Riverside – Out of Myself Darkthrone – Transilvanian Hunger Nino Rota – Amarcord Suede – Suede Darkthrone – Under a Funeral Moon

Unless you’re a huge fan of Norwegian death metal, it’s hard to see this as anything but a win for Rdio. The fact is that both services have done a tremendous job of building the celestial jukebox — with a couple of high-profile exceptions, nearly everything you’d ever want to listen to is available at your fingertips.

Now, one huge drawback of using the Rate Your Music list is that it skews towards older album-oriented music geeks. That’s great if you like Ornette Coleman and Galaxie 500, but not so great if you like Drake and Katy Perry.

Next week, we’ll set the controls for the heart of mainstream music: the Billboard charts, analyzing every charted single in the top 100 from 1955 to the present. This will give us a completely different view of their catalogs, focused on pop singles, past and present, instead of classic albums.

Want more? Ed Summers did his own fascinating deep-dive into Spotify and Rdio uses top album lists from Alf Eaton’s Album of the Year list collection, and published the results on Google Fusion Tables. Also, try Matt Montag’s Music Smasher, a tool that searches Rdio, Spotify, and Grooveshark.

(Note: This was originally published for my Wired column.)

On October 26, a YouTube user named crimewriter95 posted a full-length version of Pulp Fiction, rearranged in chronological order.

A couple things struck me about this video.

First, I’m surprised that a full-length, 2.5-hour very slight remix of a popular film can survive on YouTube for over six weeks without getting removed. Now that it’s on Kottke and Buzzfeed, I’m guessing it won’t be around for much longer.

But I was just as amused by the video description:

“The legendary movie itself placed into chronological order. If you’d like me to put the full movie itself up, let me know and I’ll be glad to oblige. Please no copyright infringement. I only put this up as a project.”

These “no copyright infringement intended” messages are everywhere on YouTube, and about as effective as a drug dealer asking if you’re a cop. It’s like a little voodoo charm that people post on their videos to ward off evil spirits.

How pervasive is it? There are about 489,000 YouTube videos that say “no copyright intended” or some variation, and about 664,000 videos have a “copyright disclaimer” citing the fair use provision in Section 107 of the Copyright Act.

Judging by his username, I’m guessing crimewriter95 is 16 years old. I wouldn’t be surprised if most of those million videos were uploaded by people under 21.

He’s hardly alone. On YouTube’s support forums, there’s rampant confusion over what copyright is. People genuinely confused that their videos were blocked even with a disclosure, confused that audio was removed even though there was no “intentional copyright infringement.” Some ask for the best wording of a disclaimer, not knowing that virtually all video is blocked without human intervention using ContentID.

YouTube’s tried to combat these misconceptions with its Copyright School, but it seems futile. For most people, sharing and remixing with attribution and no commercial intent is instinctually a-okay.

Under current copyright law, nearly every cover song on YouTube is technically illegal. Every fan-made music video, every mashup album, every supercut, every fanfic story? Quite probably illegal, though largely untested in court.

No amount of lawsuits or legal threats will change the fact that this behavior is considered normal — I’d wager the vast majority of people under 25 see nothing wrong with non-commercial sharing and remixing, or think it’s legal already.

Here’s a thought experiment: Everyone over age 12 when YouTube launched in 2005 is now able to vote.

What happens when — and this is inevitable — a generation completely comfortable with remix culture becomes a majority of the electorate, instead of the fringe youth? What happens when they start getting elected to office? (Maybe “I downloaded but didn’t share” will be the new “I smoked, but didn’t inhale.”)

Remix culture is the new Prohibition, with massive media companies as the lone voices calling for temperance. You can criminalize commonplace activities from law-abiding people, but eventually, something has to give.

Update, February 11: Everybody’s singing the YouTube Disclaimer Blues.

It’s no exaggeration to say that Occupy Wall Street first started on Twitter. As the New York Times reported Monday, the #occupywallstreet hashtag was conceived in July, a full two months before the first tent was pitched at Zuccotti Park.

As it grew from a single camp into a movement, Twitter was essential for getting real-time updates out as events unfolded, for both supporters and local government.

Particularly in the last month, some city officials have used Twitter as a tool to keep people informed. Even as they were dismantling camps, the mayors of New York City and Portland, Oregon were posting real-time updates and responding to citizens directly.

While city officials have actively communicated their positions, the response from the federal government has been muted, at best. The Occupy movement’s concerns are much larger than city politics, with most proposed demands requiring cooperation from Washington.

So far, official statements are isolated and infrequent — an early endorsement from the president, a couple of statements from the White House press secretary, and a range of opinions from individual members of Congress.

But maybe the situation’s different online? Twitter is much more casual and conversational, and social media-savvy federal agencies often respond directly to queries and complaints from their followers. It’s possible that federal employees are addressing questions and concerns about Occupy on Twitter instead.

I decided to find out.

Data Wrangling

I originally gathered this data to build the Federal Social Media Index, a weekly report that compares federal agencies using Twitter, which I’m happy to release today as part of my work at Expert Labs.

Starting with an index of over 450 U.S. government departments and agencies, I asked the anonymous workforce at Amazon Mechanical Turk to find official Twitter accounts for each one.

Three workers researched each agency, and I approved the ones they agreed on and hand-checked the rest.

When I was done, I had a list of 126 official Twitter accounts representing a wide swath of U.S. government, from the Secret Service to the Postal Service. (Browse them all on the Federal Social Media Index or in the spreadsheet below.)

To collect all the tweets, I used ThinkUp, a free, open-source tool for archiving and analyzing social-media activity on Twitter, Facebook, and Google+ that I work on at Expert Labs.

With this dataset, I could easily tell which federal agency is the most popular (NASA), the most prolific (the NEA), and the most likely to reply to you personally (the US Census Bureau).

It also makes it very easy to see who’s talking about Occupy, and who isn’t.

Occupy Silence

Since the Occupy protests started in mid-September, nearly 15,000 messages were posted by the 126 federal Twitter accounts.

Of those accounts, only three have mentioned the Occupy protests in any way — Voice of America, the Smithsonian, and the White House.

For those unfamiliar with it, VOA is a radio and television news network broadcasting in 100 countries in 59 languages, but banned from airing in the United States because of propaganda laws. As part of their daily news coverage, they’ve tweeted about Occupy nine times since the protests began. (Here’s the most recent.)

Second, the Smithsonian responded to a tweet by Complex Magazine, refuting rumors of an OWS-themed museum exhibit.

The only other mention of the Occupy protests: one tweet from the White House nearly two months ago.

Opening Up

The obvious reason for the silence is that the federal government doesn’t yet have a position on Occupy. If they haven’t issued a formal statement, blog post, or press conference, then why Tweet?

For starters, it’s a humane and natural way to open a dialogue with a generally forgiving audience. Some of these agencies have tens or hundreds of thousands of people who care about what they have to say, or they wouldn’t be following them.

Proactively talking about potentially challenging issues like Occupy is an opportunity to bring some humanity to government, and maybe even help shape policy.

(Note: This was originally published in my column in WIRED.)

I was stunned and appalled by the UC Davis Police spraying protestors, but struck by how many brave, curious people recorded the events. I took the four clearest videos and synchronized them. Citizen journalism FTW. Sources below.

Best viewed in HD fullscreen.

Top

briocloud, http://www.youtube.com/watch?v=K8Uj1cV97XQ

jamiehall1615, http://www.youtube.com/watch?v=wuWEx6Cfn-I

Bottom

OperationLeakS, http://www.youtube.com/watch?v=BjnR7xET7Uo

asucd, http://www.youtube.com/watch?v=6AdDLhPwpp4