Psychonauts is one of my favorite games of all time, the story of a psychic summer camp for kids, a government conspiracy, and love story rolled into one. The writing is funny and sweet, the characters and world wildly memorable, with some of the most inventive and innovative level design I’ve ever seen.

I first linked to the Psychonauts demo a week before its release in April 2005, and it still holds up incredibly well. If you’ve never played it, it’s deeply discounted on Steam in a $20 bundle with Broken Age, Grim Fandango Remastered, Brutal Legend, Costume Quest, and a bunch of other great Double Fine games for the ridiculous price for the next 16 hours. Or just pick it up for $10 normally.

It’s a miracle the game was ever released at all. As Double Fine’s first game, its development was notoriously fraught with major obstacles — a neverending crunch mode over 4.5 years of development, cancelled funding, and multiple near-death experiences. In the end, the game was released to critical acclaim and relatively weak sales, but built a deep and obsessive cult following over the last decade, eventually selling 1.7 million copies.

To commemorate the launch of their new campaign to fund its sequel, Double Fine and 2 Player Productions made a 50-minute short about the history of Psychonauts. It’s an incredible archive of Tim Schafer’s unseen video footage, design thinking, and interviews with the team talking about their experience making the game, good and bad.

Double Fine, more than any indie studio I know, is defined by its willingness to keep pushing itself to try new things. Some of these experiments work and some fall flat, with other indies able to watch and learn from the sidelines.

Their Kickstarter project for Broken Age paved the way for many other indie game developers and fans to use crowdfunding, and they documented it all in public with the Double Fine Adventure documentary, one of the best accounts of making anything ever, entirely free on YouTube. Their Amnesia Fortnight project, first internal and later public, encouraged experimentation within a team. Their Steam Early Access projects are perhaps the most controversial, with the cancellation of one game, though even that’s found a life of its own.

And, now, with the Psychonauts 2 campaign on Fig, they’re testing the waters for equity crowdfunding: the ability for members of the public to get a financial return on a crowdfunding project, rather than rewards alone.

(As a Kickstarter advisor and shareholder, I’m biased, but I have concerns about equity crowdfunding for indies. If you think backers feel entitled now, wait until they’re expecting a financial return.)

But I admire Double Fine for pushing everything forward, and I’m excited to see the results of their experiment. I supported the Double Fine Adventure, and of the 229 projects I’ve backed on Kickstarter, it’s still my personal favorite. I have my signed poster, backer shirt, and I played and loved Broken Age. But it was worth it for the documentary alone, seeing every exciting and painful moment of making the game with the people behind it.

So, of course, I backed the Psychonauts 2 project and I can’t wait to see what happens. If you want to come along for the ride, you can still back the project until January 12.

As a side note, I played a teeny, tiny role in this story.

Five years ago, when Kickstarter was only a year old and two years before Notch made headlines with a tweet, I read this Joystiq article by Justin McElroy, writing that Tim Schafer was open to making Psychonauts 2 if he could find a willing publisher.

I freaked out, asked my friend Brandon Boyer for an email intro, and sent this email to Tim Schafer on November 12, 2010.

Tim: Huge, huge fan of your work. I recently replayed Psychonauts, DOTT, and MI 1 and 2 with my six-year-old son. Just as good as I remembered.

Anyway, I showed the Joystiq article to the Kickstarter team and we’re all freaking out about the prospects of what Psychonauts 2 on Kickstarter might look like. Jamin from Kill Screen’s at the Kickstarter office right now and they’re all talking about it.

Going directly to fans to pre-sell the game before it exists sounds insane, but it’s very possible. It’s an incredible promotional tool, a great way to show there’s a market for the game to publishers, and doesn’t involve any kind of investment or preclude any kind of future publisher arrangements. And, of course, Kickstarter would promote the hell out of it to the community.

I’d love to introduce you to the team and answer any questions you might have, if you’d even *remotely* consider this.

I never heard back, but three years later, I invited Tim Schafer to speak at XOXO. There, for the first time, I heard the story of what happened after he received my email, the conversation that it kicked off internally at Double Fine with his business manager, and how it eventually led to the record-breaking Double Fine Adventure project. (The story starts at 17:30.)

Back in October, NPR’s Hidden Brain podcast covered the story of professional poker player Annie Duke, the only woman to compete in the World Series of Poker: Tournament of Champions in 2004.

It’s a fascinating story about her own impostor syndrome, feeling like she didn’t belong at that table, but also how she used gender stereotypes to work in her favor and eventually win the competition.

I love everything about this story. Once you’ve listened to it, I recommend watching the final moments of the game on YouTube.

For another perspective, Wil Wheaton pointed me to Annie Duke’s talk on The Moth, a gripping retelling in her own words of that tournament. It was the first time she’d ever played on television, and the first time anyone could see her hand.

While listening to the NPR story, I went looking for a refresher on the rules of Texas hold ’em, and ended up on this Wikipedia page for betting in Poker.

Immediately, I was struck by the language, which was dominated by male pronouns:

When it is a player’s turn to act, the first verbal declaration or action he takes binds him to his choice of action; this rule prevents a player from changing his action after seeing how other players react to his initial, verbal action.

If he declines to raise, he is said to “check his option.”

If a player borrows money to raise, he forfeits the right to go all-in later in that same hand — if he is re-raised, he ”must” borrow money to call, or fold.

And so on, for over 12,000 words. It felt like it was borrowed from another time, cribbed from a thrift shop poker book from the 1970s.

Because it was Wikipedia, I felt like I could do something about it. So I spent some time making the biggest edit I’ve ever made on Wikipedia: changing every male pronoun to gender-neutral language, sometimes rephrasing as “the player,” but often using the singular they. I tried to be careful about readability, making sure to only use it in cases where it couldn’t be confused with a plural group.

So “a player may fold by surrendering his cards” became “a player may fold by surrendering their cards.”

In the end, it took longer than I’d like to admit — over 160 changes in one big commit.

I saved it, tweeted about it, and promptly forgot about it.

While listening to @AnnieDuke on NPR yesterday, I looked up some poker rules on Wikipedia. All male pronouns. So… https://t.co/VifkHy4dPQ

— Andy Baio (@waxpancake) October 9, 2015

Yesterday, I remembered the change and popped over to Wikipedia to see if it survived.

Unsurprisingly, every change was reverted less than a week later. The user left a reason: “‘they’ is a plural term and inappropriate for an encyclopedia article.”

As it turns out, Wikipedia has its own guidelines about gender-neutral language. The Manual of Style recommends, “Use gender-neutral language where this can be done with clarity and precision. For example, avoid the generic he.”

A Wikipedia essay expands on the guideline, “There is no Wikipedia consensus either for or against the singular they… Although it is widely used in informal writing and speech, its acceptability in formal writing is disputed.”

Fortunately, that’s changing.

The singular “they” is one of the most hotly-debated subjects in exciting world of grammar. Last month, Dennis Baron declared it the “word of the year.”

Just last week, the Washington Post style manual became the latest to accept the pronoun. Bill Walsh, The Post’s copy editor, explains their decision:

There was one change, though, that I knew would cause controversy. For many years, I’ve been rooting for — but stopping short of employing — what is known as the singular they as the only sensible solution to English’s lack of a gender-neutral third-person singular personal pronoun. (Everyone has their own opinion about this.) He once filled that role, but a male default hasn’t been palatable for decades. Using she in a sort of linguistic affirmative action strikes me as patronizing. Alternating he and she is silly, as are he/she, (s)he and attempts at made-up pronouns. The only thing standing in the way of they has been the appearance of incorrectness — the lack of acceptance among educated readers.

What finally pushed me from acceptance to action on gender-neutral pronouns was the increasing visibility of gender-neutral people. The Post has run at least one profile of a person who identifies as neither male nor female and specifically requests they and the like instead of he or she. Trans and genderqueer awareness will raise difficult questions down the road, with some people requesting newly invented or even individually made-up pronouns. The New York Times, which unlike The Post routinely uses the honorifics Mr., Mrs., Miss and Ms., recently used the gender-neutral Mx. at one subject’s request. But simply allowing they for a gender-nonconforming person is a no-brainer. And once we’ve done that, why not allow it for the most awkward of those he or she situations that have troubled us for so many years?

Grammar manuals and copy editors may be slow to adapt to how the rest of the world uses language, but the increasing popularity of “they” reflects an increasingly gender-inclusive culture.

In the meantime, Wikipedia leaves the singular “they” in limbo, neither endorsed nor banned. Most of the arguments seem to boil down to some variation of “it looks ugly.”

But what’s uglier: a mismatch of number or a mismatch of gender? One is mildly irritating, maybe slightly confusing. The other is often insulting and alienating.

Language evolves, and there’s little prescriptivists can do to change it.

Eventually, they’ll have to suck it up, accept the cards on the table, and fold.

Though they’ll inevitably make a lot of noise in the process.

Two weeks ago, the server I host all my personal projects on was hacked by some guy in Ukraine. It really sucked.

I was overdue for a redesign anyway.

I first noticed something was amiss while trying to post a link here and the server was unusually slow. I SSHed in and the server was slow to respond, as if system resources were being consumed by a runaway process.

A quick ‘top’ revealed that MySQL was pegging the CPU, so I logged into the MySQL console and saw that a dump of the database was being written out to a file. This was very unusual: I never schedule database backups in the middle of the day, and it was using a different MySQL user to make the dumps.

Then I noticed where the mysqldump was being written to: the directory for a theme from a WordPress installation I’d set up the previous month, an experiment to finally migrate this blog off of MovableType.

This set off all my alarms. I immediately shut down Apache and MySQL, cutting off the culprit before they could download the dumped data or do any serious damage.

I’d recently updated to the latest WordPress beta, and saw that the functions.php file in the twentysixteen theme directory was replaced with hastily-obfuscated PHP allowing arbitrary commands to be run on my server through the browser.

This confirmed all my lingering unease about running WordPress, built up over a decade of hearing horror stories of friends and acquaintances getting hacked–but that stereotype of WordPress security was outdated and wrong, and led me to make a very stupid, very serious blunder.

I moved the WordPress install, along with the hacked PHP and aborted mysqldump, to my local machine and deleted it from my server. I looked through the logs to see what else they’d been up to, and convinced I’d closed the hole by removing WordPress, eventually started my server back up to minimize downtime.

The next day, “Ivan” dropped every database in MySQL, deleted my blog, and replaced it with this pseudo-political polemic he’s used on other compromised sites.

(As an aside, the embedded YouTube video is this dubstep remix of the Requiem for a Dream theme by Clint Mansell and Kronos Quartet, misattributed to Hans Zimmer. Your guess is as good as mine.)

How It Happened

After going through every log file, and with the help of Gary Pendergast from the WordPress security team, I assembled a minute-by-minute timeline of what happened.

Our friendly hacker first appeared in the logs on the Waxy.org homepage, running a vulnerability scan testing thousands of different URLs to find possible vectors of attack. And it finds one, a copy of PHPMyAdmin that I apparently installed in 2002 and forgot about it entirely.

He tried to sign in briefly, but failed, so starts looking for other PHP scripts on the server using a simple Google query for “site:waxy.org inurl:php”. This turns up half a dozen results, with one that looks promising — a project I did in 2005 to visualize a data dump that Boing Boing released to commemorate their fifth birthday.

He starts an open-source toolkit called SQLMap to probe the script for SQL injection holes, it quickly finds one, and uses it to own the database.

In the database, he sees a database for WordPress from the installation I mentioned earlier. He fires up a third vulnerability scanner called WPScan to search for WordPress vulnerabilities, but it’s not clear if he finds any.

Either way, it’s not necessary — with access to MySQL, the culprit can add himself a WordPress admin and sign in. Immediately, he uses the WordPress theme editor to install malware PHP to the theme, allowing him to execute arbitrary commands on the server. Just in case, he writes copies of the malware PHP to three more locations outside of the WordPress installation in case it’s deleted.

So, after I removed access to WordPress, he was still able to get to the malware needed to own the box. Eventually, he grows bored and deletes the database and everything on Waxy.org.

Comedy of Errors

Fortunately, I had a database backup from earlier that morning, and a recent backup of all files. I killed all services on the server, and started the long process of restoring sites carefully, one by one, with modern security practices in mind.

But this was easily one of the most miserable, stressful experiences of my life. Yesterday, I woke up in the middle of the night with a cold-sweat nightmare that I was hacked again.

I had a PTSD-ish nightmare that my server was hacked again, this time from an exploit in Postfix. Stupid lingering stress.

— Andy Baio (@waxpancake) December 9, 2015

And it was so avoidable, born from laziness and complacency. Let’s go through the highlights of bad security practices:

1. My old server at Softlayer was running continuously for eight straight years, since December 2007, and there was code carried over from previous servers dating back to 2002.
2. The Boing Boing Stats was a throwaway PHP hack that sat untouched for a decade on multiple servers with a glaring SQL injection hole. And, hell, I didn’t even know that ancient copy of PHPMyAdmin installation was still hanging around.
3. I was using a shared MySQL user account for nearly every project running on the server, which had near-universal permission to delete records or drop databases entirely. Plus, it allowed for remote connections. So bad.
4. I played loose with file permissions, giving the Apache user the ability to write to far more than it should have.
5. I was running Centos 5, but not keeping up-to-date with security updates.
6. Critically, I wasn’t running any software to monitor and ban vulnerability scans or alert me to malicious activity.

And that’s just scratching the surface of issues relevant to this hack. I was still using password-based logins with SSH, root logins were available, MySQL passwords were weak… Frankly, it’s amazing I wasn’t hacked earlier.

Righting Wrongs

If there’s a bright side to any of this, it’s that it gave me a long-overdue crash course in modern infosec practices. And migrating from a dedicated leased server to virtual servers feels like waking up in the future.

After a bunch of research, I decided to abandon dedicated servers entirely and move to a beefy DigitalOcean droplet running Ubuntu 14.04. It’s more powerful than my old server, provisioned instantly, and I’m paying a fraction of the price. DigitalOcean’s admin tools are phenomenal, and backups are automatic and painless.

DigitalOcean’s tutorials are absolutely incredible, and I found them invaluable in initial setup, securing Ubuntu, my firewall, MySQL, and using Fail2Ban to protect Apache and SSH. There’s still more work to do for monitoring intrusions, but it’s a start.

So, all of that sucked. But, while bittersweet, I’m better and stronger for it.

Thanks, “Ivan.”

I’m so excited about this, I had to try it myself—Jason Scott announced that the Internet Archive now supports embedding playable games and other software from their collections in web pages. (Look for the share icon on any software page.)

Continue reading “Embedding Software History” →