Ben Kamens

Lead dev at Khan Academy
and proud past at Fog Creek

Margaret Hamilton and the First Men on the Moon

First Men on the Moon is one of my favorite little educational resources. It’s the closest I’ve ever felt to being part of the Apollo 11 moon landing. They took the actual video filmed outside the window of the lunar module, the audio recorded between astronauts and mission control, and a bunch of data about the landing itself — including Armstrong’s heartrate over time — and synced it all together into a sweet interactive experience.

Looks something like this, ‘cept on the site you hear everything and can scrub back'n'forth from descent to “The Eagle has Landed!”

Playing w/ this site is how I learned about a software glitch that almost botched the whole thing. Neil Armstrong is a couple minutes from touching down when he says “program alarm” so calmly you could almost miss it amid the chatter. Goes something like:

Armstrong: "Program alarm."

[4 seconds, unrelated chatter]

Armstrong: "It's a 1202."

Aldrin: "1202."

[16 seconds of silence]

Armstrong (now with urgency in his voice): "Give us a reading on the 1202 program alarm."

Mission control: "Roger. We got...we're go on that alarm."

And on the Mission Control loop in Houston — the one the astronauts can’t hear — they say:

Steve Bales: "If it doesn't reoccur, we'll be go."

Go listen to it! For those of you who haven’t spent your lives in software, let me translate from Super Professional NASA Speak™ into ordinary developer chat for you:

This is obviously a silly caricature. NASA are professionals. But I read about this event after listening to the audio. Turns out there’s a bit of truth here — when that 1202 error first showed up, there was a lot of confusion. Only one person, thanks to a previous simulator failure, happened to have a hand-scribbled list of various possible program alarms and was able to suggest that it was safe to keep going. The alarm would pop up three more times before landing.

It’s always been one of my favorite moon-landing stories — almost aborting right at the very end.

So I was really happy to learn about the programmer behind this success: Margaret Hamilton. I’d never heard her name until a recent reddit thread and a followup on hacker news.

Turns out she’s not only responsible for this alarm not ruining the mission but she actually coined the term 'software engineering’. Apparently she spent a lot of time arguing for software safety checks that prevented astronauts from accidentally doing the wrong thing, despite being regularly assured that this code wouldn’t be worth its weight because professional astronauts don’t accidentally do the wrong thing.

In Apollo 11’s case, a switch in the lunar landing module was set in the wrong position, causing its computer to process data from two radars instead of just the one pointed at the moon. This was too much to process, so Hamilton’s system began (impressively) dropping its low priority tasks in order to have enough CPU to properly handle the critical lunar surface radar input. Wasn’t a software glitch at all — quite the opposite. A graceful handling of a critical situation.

Rest is history. Enjoy it for yourself, and be sure to thank Margaret when Neil gets to ignore those scary-sounding alarms.

A book hiding in a gist

A while back — after leaving Heroku — Adam Wiggins left this little note on twitter:

I normally don’t just “reblog” someone’s work, but this humble little gist just packs so much wisdom so tight. Anyone building a product team will love the quick read.

Me? I find myself re-skimming these gems every few months. “Make it real.” “Ship it.” “Do it with style. ”Own up to failure.“ ”Write well.“ I like keeping these in mind while we build our team at Khan Academy.

Each little bit in that gist could easily be its own blog post. Altogether there’s a great book hiding.

KA Lite

Choosing what to work on is one of the hardest things we do at Khan Academy. Saying no’s hard for everyone, but it sure does feel tough when the person you’re saying no to is some earnest child trying to learn.

Two of the hardest no’s we’ve ever said were aimed at students who couldn’t speak English and students who don’t have access to the internet.

For a long time we said no to internationalization. i18n is hard. Especially when you have more content than all the Harry Potter books combined, much of it in video form. We were a small team. With a core product to figure out and without the time to make it work in other languages. We’d hear cries from all over: “my daughter needs help but doesn’t speak English,” “my students are dying to use KA but can’t navigate the site,” “we’ll translate everything for you, just please tell us how!” We always knew we’d bite the bullet one day, but for a long time we just had to say no, no, no.

At some point the chips fell into place. And now the whole shebang’s available in Spanish, French, Turkish, Portuguese, and soon many others. Looking at those pages makes me proud. The thought of not-just-English-speakers getting access to free educational content reminds me why we’re doing what we’re doing.


“No.”

Unfortunately, those without internet are in a different ship altogether. Despite the fact that any student w/out internet is probably also a student most in need of free educational resources, we simply can’t yet focus on that Ridiculously Hard Problem.

Enter Jamie Alexandre, a Summer 2012 intern at Khan Academy. During our first ever hackathon Jamie hacked a Raspberry Pi to run a modified version of KA entirely on cheap components without needing any access to the internet. Fast forward a few months, and next thing we know he tells us he’s started his own non-profit, the Foundation for Learning Equality, dedicated to bringing their own custom version of KA’s platform (“KA Lite”) and other online educational resources to everybody — especially those offline.

I love this story for about a million and twenty reasons. Selfishly it’s just so cool to see an incredible and entirely separate organization spring out of our internship. Non-selfishly, KA Lite has now been installed in over 120 countries, giving access to students, teachers, and even inmates who don’t get internet and all that comes with it.

Internally we still think the best way Khan Academy itself can improve access for those without internet is to build the best educational product and content we can — doing so will motivate others to help us reach everyone one day. But it sure is a lot easier to say no to kids without internet when Jamie and his crew are tackling the problem in their own way.

Thanks for all you do, KA Lite. Please come back and visit for our next healthy hackathon!

Going over the top

When Andy recently decided to join Khan Academy, he emailed us a secretive app and said to run it on an iPad. When we did we found a fun little toy — if the device was angled in just the right way and the little knobs were in just the right position, the words “I accept” playfully emblazoned themselves on the screen. It was such a cool way to join the team.

Point is, he could’ve just emailed us his signed offer letter and the team would’ve been thrilled. Honestly he could’ve used however many hours he spent on that app to get “real work” done or find a great pizza spot. Nobody would’ve thought twice.

When we recently ran the third Khan Academy Healthy Hackathon, I announced it with a ridiculous Doctor Seuss poem (copied below in case you haven’t rolled your eyes today). I also took the time to make a whole little single-page site dedicated to introducing the hackathon.

Point is, all of that was completely unnecessary. Our hackathon was internal only and a quick email to team@khanacademy.org would’ve saved hours. There were at least twenty-seven better ways I could’ve been spending my time.

But if every single person made every single decision based only on what’s most valuable at that exact moment, we’d always be fighting fires.

I’ve come to appreciate how important it is for an organization to make it safe for folks to go over the top. To express themselves by taking an ordinary task — one that you might think isn’t important — and investing so much passion into it that people say, “that’s nuts.” I wanna work with more of those nuts. And I wanna help build a team that embraces ‘em.




You have hacks in your head
But never the time
The last thing you need
Is to be reading a rhyme.

But lo and behold, an opportunity rises
To hack hacks and make makes and prize all sorts of prizes.
It's the third time around and a chance for some firsts
Hackathon Healthy Academy Khan — now read that reversed.

Some hacks will work well
They'll change lots of lives
We'll ship 'em bug-free
And exchange five high-fives.

Except when they don't.
Because, sometimes, they won't.

I'm afraid that some hacks
Might not even ship
That's ok! Time for fun!
Hack make build, let 'er rip.

Three minute quiz: App Engine datastore performance

As somebody now spending all his time in NoSQL land, my brain perked up when working through The Three Minute SQL Performance Quiz. It was a blast from the past for me, a chance to remember all the little ins'n'outs of SQL performance from a quaint old time when I used to be able to write JOIN statements.

So I thought it’d be fun to develop a similar performance challenge for this new NoSQL life of mine. Since we use App Engine at Khan Academy, we’ll focus on the App Engine datastore. Proceed.

Question 1


QueryModel
Monkey.all().filter(
    "genus IN",
    ["Ateles", "Cebus", "Aotus"]
  ).fetch(10)
class Monkey(db.Model):
  genus = db.StringProperty(
    indexed=True)

Hold on there — a major improvement is possible or All looks good to me — don’t go changin’ a thang

You’re right! Queries that use the IN operator may look like a single query, but they actually run multiple queries behind the scenes, one for each item in the list. That means if you ran the above query, opened Appstats, and looked at this request’s profile, you’d see this:

…and that’s not ideal if you really care about this request’s performance. The requests are running asynchronously and overlap as much as possible — which is great — but you’re still running three requests and increasing the likelihood that one of ‘em will slow you down.

If you want this to be blazing fast, you almost certainly want to denormalize this set membership into a property that can be queried without an IN operator. Perhaps you’d add something like is_in_favorite_genus = db.BooleanProperty(indexed=True) to Monkey, set that property to True if the genus is one you’re interested in, and then change your query to Monkey.all().filter("is_in_favorite_genus =", True). That’d be a significant improvement — especially if your IN list contained many items. That’s the App Engine way.

Question 2


QueryModel
Monkey.all().filter(
    "unique_name =",
    "bob_the_monkey"
  ).get()
class Monkey(db.Model):
  unique_name = db.StringProperty(
    indexed=True)

Hold on there — a major improvement is possible or All looks good to me — don’t go changin’ a thang

You’re right! If you’re loading a single entity using a unique identifier, queries aren’t as fast as loading by key or key name. There are multiple ways to load an entity by key that we won’t get into here — just know that if you have unique identifiers for your models, you should strongly consider constructing your entities using the unique identifier as part of your model’s key name — Monkey(key_name="bob_the_monkey", **kwds) — so you can quickly retrieve it later: db.get_by_key_name("bob_the_monkey")

If you’re curious why this is faster than a query, it makes sense if you’re willing to swallow the gross oversimplification that a query is just quickly looking up the key in an index and then fetching the entity using the key. If you’ve already got the key, why query?

Question 3


One of the coolest dogs ever or Meh, she’s so-so

You’re right! Shouldn’t need explainin’.

Question 4


Putting an entityModel
m = Monkey(
    name="Bob the Monkey",
    favorite_color="blue",
    favorite_food="pizza",
    worst_enemy="honey_badger"
  )

m.put()
class Monkey(db.Model):
  name = db.StringProperty()
  favorite_color = db.StringProperty()
  favorite_food = db.StringProperty()
  worst_enemy = db.StringProperty()

Hold on there — a major improvement is possible or All looks good to me — don’t go changin’ a thang

You’re right! Kinda. It depends how you’re querying this entity. All of the datastore properties on Monkey are indexed by default, and that means every time you put() a Monkey, you’re spending time writing new values to all of those indexes.

Now, if you need to be able to run queries to find Monkeys based on their favorite_color, favorite_food, worst_enemy, and so on — then you’re doing the right thing. You need the indexes to be able to query them. If your code base is anything like Khan Academy’s used to be, however, you may have tons of properties w/ automatic indexes that never, ever need to be queried. We fixed this by using db.StringProperty(indexed=False) in these cases (and writing a linter that requires us to explicitly specify indexed=True|False for every datastore property).

Write speed may not matter for your app, but if it does you can speed up writes by not wasting time writing to indexes you don’t need.

Question 5


QueryModel
query = Monkey.all()
query.filter("name =", "bob")
query.filter("zoo =", "Manhattan")
query.filter("hats >", 5)
query.get()
class Monkey(db.Model):
  name = db.StringProperty(indexed=True)
  zoo = db.StringProperty(indexed=True)
  hats = db.IntegerProperty(indexed=True)

Hold on there — a major improvement is possible or All looks good to me — don’t go changin’ a thang

You’re right! This query will work. And when you first start out and only have a couple hundred Monkeys, it’ll be blazing fast. But as your family of Monkeys grows, you may find yourself with a Very Serious™ performance issue. Why? You have all of the properties indexed, you’re only asking for a single entity…what’s the problem?

You don’t have a perfect index defined. Without an index that perfectly covers all of the properties involved in the query’s filters, here’s what App Engine has to do when you ask it to fetch a single entity: start querying the datastore for, say, Monkeys with name == “Bob”, retrieving them from the datastore, and filtering through them to find any with zoo == “Manhattan” and at least 5 hats. That’s not what you want, and it looks like this in Appstats:

That’s a disaster. Depending on the shape of your data, you could spend hundreds of RPCs just returning a single entity from the datastore. You want App Engine to query the datastore for “Monkeys named Bob in the Manhattan zoo who own at least 5 hats” in one swift, single blow — it should only need one RPC. And it’ll do exactly this if you have a perfect index. You’ll need an entry in index.yaml, something like:

- kind: Monkey
  properties:
  - name: name
  - name: zoo
  - name: hats

If you aren’t careful and don’t keep an eye to make sure that your important queries have indices that perfectly cover 'em, you could have a piece of code that runs snappy one day and, months later when your datastore fills up with Monkeys, it’ll suddenly take 5+ seconds to return a single entity. We’ve experienced this at Khan Academy more than once. It’s public enemy number one due to how easy it is to not notice the problem at first and then encounter brutal performance problems later.

Question 6


QueryModel
query = Monkey.all()
query.filter("name =", "bob")
query.filter("zoo =", "Manhattan")
query.filter("hats >", 5)
query.get()
class Animal(db.PolyModel):
  name = db.StringProperty(indexed=True)
  zoo = db.StringProperty(indexed=True)

class Monkey(Animal):
  hats = db.IntegerProperty(indexed=True)
 index.yaml
 
- kind: Animal
  properties:
  - name: name
  - name: zoo
  - name: hats

Hold on there — a major improvement is possible or All looks good to me — don’t go changin’ a thang

You’re right! What now?! You’ve got your perfect index on all the queried properties! Well, not really.

This’ll suffer from the exact same Very Serious™ performance issue from Question 5. By using PolyModel and querying specifically for Monkeys — not just any old Animal — you’re implicitly adding another filter to your query. This filter will use PolyModel’s built-in special property, class, to only return Monkey results. If you don’t have an index that covers all filtered properties including class, you ain’t go no perfect index.

- kind: Animal
  properties:
  - name: class
  - name: name
  - name: zoo
  - name: hats



That’s it, you’re done! I could keep going, but by now you’ve spotted a trend in the answers and I’m pretty sure we’re past the 3-minute mark.

If you want more or have other interesting quiz questions I’d love to know. And if you geek out on perf work like me, you know what to do.

Email transparency at Khan Academy

Whenever we mention that almost all Khan Academy email is visible to everybody on the team, people always wanna know more.

The idea is unapologetically copied from Stripe. Whether they originally came up with it or not I dunno (edit: they did). We certainly didn’t. But by now we’ve added enough of our own little tweaks to warrant contributing back. Here’s the how and why of “radical email transparency” at Khan.

How we got started

  • Step 1) Read Stripe’s post.
  • Step 2) Get forwarded Stripe’s post by at least 3 other devs within a couple days.
  • Step 3) Implement a super hacky version of Stripe’s post

That all happened ~10 months ago. I’d absolutely do it again. I’m not gonna list all the ways this is different than Stripe’s — our system is just a bit simpler/smaller/stupider.

How our hacky version works

Every team has two email addresses: one for team members and one for the team’s “blackhole.” analytics-team@khanacademy.org and analytics-blackhole@khanacademy.org.

The -team@ address is for emailing all members of the team.
When you send email to analytics-team@, you expect everyone on the analytics team to read it.
Subscribing to analytics-team@ means analytics-related email will land in your priority inbox as soon as it’s sent, and you’re expected to read it.

The -blackhole@ address is for anything else that has anything to do with analytics.
When you CC:analytics-blackhole@, you don’t expect subscribers to immediately read it.
Subscribing to analytics-blackhole@ means you’ll receive analytics-related email, but it’ll get filtered out of your inbox and you’re not expected to read it unless you feel like it.

There’re two additional catch-all lists for the entire dev team: dev-team@ and dev-blackhole@. So we currently have ~24 lists. dev-team@, dev-blackhole@, analytics-team@, analytics-blackhole@, mobile-team@, mobile-blackhole@, i18n-team@, i18n-blackhole@, …

Anybody in the org can join any of these email lists. analytics-team@ is usually just team members, but analytics-blackhole@ has all sorts of lookie-loo subscribers who’re interested in analytics happenings.

All email to analytics-team@ gets forwarded to analytics-blackhole@, so lookie-loo -blackhole@ subscribers will get updates sent to -team@ automatically.

When you receive an email via a -blackhole@ list, it is automatically tagged as blackhole and filtered out of your inbox.

If an email is directly addressed TO:you@ or TO:your-team@, it’ll stay in your inbox regardless of any CC:-blackhole@.

Best practice when sending email: unless you have good reason not to, CC dev-blackhole@ or any of the other -blackhole@ lists.

Best practice when reading email: read the emails in your blackhole on your own schedule, if and when you want.

Between these buckets and best practices we have the tools we need to make the contents of any email open to anyone interested without burying ourselves in a deadly deluge.

Example emails

  • Jace emails Ms. Monkey (an analytics teammate of his) about a new experiment being run. TO:msmonkey@, CC:analytics-blackhole@
  • Tom emails the internationalization team about a change that may affect the way usage statistics are calculated for non-English users. TO:i18n-team@, CC:analytics-blackhole@
  • I email Mr. Gorilla asking when he’d like to demo his latest work for the company. TO:mrgorilla@, CC:dev-blackhole@
  • Craig emails the infrastructure team with a summary of upcoming priorities. TO:infrastructure-team@
  • I email Marcia about a personal career matter that shouldn’t involve others. TO:marcia@.

How it’s technically implemented

Google Groups ‘n’ Gmail filters. You’re about to see how hacky the rabbit hole goes.

Google Groups

  • Each team gets their own -team@ and -blackhole@ groups.
  • Each -blackhole@ group is a member of its respective -team@ group.
  • Group settings:
    • Permissions | “Who can join this group?” ==> “Anyone in the organization”
    • Description includes “(email-transparency)”
    • Information | Directory | Check “List this group in the directory”
  • We setup a short URL to a google groups search for “(email-transparency)” — now anybody can go to khanacademy.org/r/email-transparency and subscribe to whatever lists they want.

Gmail Filters

  • Everybody has the following filters. Our setup doc has 'em exported to xml, which everybody imports via Gmail | Settings | Filters | Import filters.
    • Matches: to:(*-blackhole.khanacademy.org)
      Do this: Apply label “blackhole”
    • Matches: from:(-me) to:(*-blackhole.khanacademy.org -me -*-team.khanacademy.org)
      Do this: Skip Inbox
  • Now emails received via blackhole lists are labeled and filtered out of your inbox unless they were addressed to you or your team.

Why I’m happy

This experiment pushed my personal comfort zone at first — “wait, is it really ok for everyone to see this?” — but I feel great about an email culture that’s open by default and wouldn’t wanna go back.

Trusting devs to subscribe to whatever content they find helpful makes me happy. Trusting everyone to not waste time reading email that’s unimportant for them has worked out. We’ve seen a lot of value from -blackhole@ subscribers getting news they wouldn’t otherwise have seen.

You know those moments right before you send an email when you sit there and type names into CC, then delete 'em, then re-type 'em, all because you’re trying to figure out who cares? That’s gone. It’s up to the subscribers. Just send it to the critical people and CC a blackhole list. Fewer email decisions.

You know those once-in-a-while emails that’re full of insight? Bullet-point priority breakdowns and strong opinions and team schedule updates? Sometimes they’re only read by one or two people, maybe because the author doesn’t feel comfortable blasting all@company.com. That’s just silly. I like 'em in an open, searchable spot. Blackholes are perfect.

You know how your inbox is already overloaded and the last thing you want is more email? No problem, don’t subscribe. Or only subscribe to -blackhole@ lists and only skim 'em once in a blue moon. I do the latter.

You know how you keep saying you want to keep your team as flat as possible for as long as possible? Avoiding titles is one thing, sure. But flattening communication — trusted access for everybody — is huge.

Openness and trust has been a big part of Khan Academy’s dev team since the start. This experiment kinda fit right in.

Why I’m not so happy

Our biggest problem, by far, comes from imperfections in the gmail filter setup. Filters have no way of knowing which lists you’re subscribed to, so if somebody sends an email to analytics-team@ and dev-blackhole@ and you receive the email, we don’t know if it should stay in your inbox or not. You may’ve received it via the blackhole list, but if you’re subscribed to analytics-team it shouldn’t disappear to the blackhole.

Right now we err on the side of safety by not removing anything from your inbox if it was sent to a -team@ address, but this means some messages that should be in your blackhole are not. Stripe has tried to work around this via a library to automatically generate complicated gmail filter combinations, but that doesn’t feel quite right for us. It’s a bit complicated for newcomers to use, and it requires reconfiguring your filters every time you join or unjoin a new list.

I’m sure there are tools out there that could solve this — probably by abandoning google groups altogether. Ideas welcome.

I’m also not in love with the google groups UX that lets people find/join/unjoin lists: khanacademy.org/r/email-transparency. No question there are tools that’d handle this use case better. Supposedly Stripe built a custom interface on the Groups API.つ ◕_◕ ༽つ giff open source plz!

Anybody else on this train?

Have others experimented w/ email transparency? We won’t go back. But as you now know our setup is far from perfect. Would be interested in learning from others.

The most common feedback we give dev interns

I’ve been lucky to see ~70 interns pass through the dev teams at Khan Academy and Fog Creek. If you know me you know how much I enjoy internships. Infinitely better than an interview, enough time to get meaningful work done, a chance to sit side-by-side for months, and interns see real personal and career growth (not to mention compensation).

And we’ve been lucky, talent-wise. Age be-damned, we always wind up learning from our young bloods and wishing some would never leave. But we also invest a lot of energy mentoring them, and I’d like to share the two pieces of feedback that’re most often in our mentors’ mouths. Figured this’d be useful to future interns anywhere.

1. Write code with your code reviewer in mind

We love code reviews. I’m kinda tempted to setup a picture frame on each intern’s desk, with a photo of their code reviewer kindly smiling at them and “Be nice to your reviewer!” written along the bottom of the frame in fancy cursive font.

Avoid dumping massive blobs of code on folks. You’re probably hyper-productive as an intern with nothing else to do but write code. Don’t quietly assemble a big bomb of code over 5 days and then drop it on your poor, unsuspecting team lead. Granular checkin points — one conceptual change per commit — give your code reviewers a chance to engage easily and offer feedback as you go along.

Use TODOs liberally. TODO(intern bob): transition to new API once backfill is done gives your reviewer a sneak peek inside your head. You won’t be able to do everything you want in every changeset, especially if you’re focused on shipping and following our advice for granular commits. So throw these suckers around with pride — heck, one of our past interns wrote a script that generates a TODO leaderboard from our codebase.

2. Talk about what you’re doing and how you’re gonna do it

After all those interns, I can count on half a hand the number who talked too much about what they were building.

Your work does not need to be complete to be worth sharing. Bums like me put a lot of effort into building communication channels for the team. Use them. At Khan we’ve got hipchat, transparent emails, weekly demo opportunities, dogfood days…all just begging for interns to post sketches, screenshots, design docs, and prototypes. The best communicators I know consistently face their fears by posting works-in-progress before being asked to do so (and clearly stating that more is on the way).


An excellent communicator

By the way, you’re not bragging about what you’ve done to “take credit” in some ugly political fashion. It’s part of your job to brag, to make sure your team knows your progress and what direction you’re heading.

The myth of the genius programmer

That’s a short list, I know. I wanted to title this post “You’ll Never Believe These Two Simple Tips That’ll Make You a More Valuable Intern,” but then I would’ve had to throw myself out a window and I’m pretty excited to see how 2014 turns out.

The truth is both of these pieces of feedback are about dispelling the myth of the genius programmer. Brian Fitzpatrick stopped by Khan Academy a while back to shed some light on this:

The ultimate geek fantasy is to go off into your cave and work and type and code and then shock the world with your brilliant new invention. You know, it’s a desire to be seen as a genius by your peers.

The fantasy — the myth that feverishly creating while all alone is genius — hits all of us, especially interns coming right out of school. It’s certainly lodged somewhere deep inside me. But Fitz and Ben know that only emerging from your cave when you’ve discovered perfection is insecurity, not genius. It’s what makes us accumulate huge unreviewed swaths of code and clam up when we should be sharing our progress.

The best programmers I know brave their imperfections. They limit themselves to granular changes that others can understand and constantly talk about whatever they’re doing. And that’s why we give interns the above advice. If you can somehow show up and apply your talents in this same way right out of school, you’re probably gonna be seen as a genius.


We’re still accepting applications for our next class of Khan Academy interns.

Due props to Tom Yedwab and all other mentors who directly or indirectly contributed.

“Shipping beats perfection” explained

When we sat down to write Khan Academy’s company values in 2010, “shipping beats perfection” flew out of Sal’s mouth before our butts hit the chairs.

It’s almost 3 years later. I have one-on-ones with teammates new and old who wanna talk more about what “shipping beats perfection” means. When outsiders happen across our development principles, “shipping beats perfection” becomes a lightning rod for anything from compliments and respect to accusations and fury. Our internal chat bot, the culture cow, drops “shipping beats perfection” as occasional chat room reminders (between MOOs).

At some point along the way when the phrase passed between my ears for the NNth time, my brain started believing it was invented by Google or Facebook (nope, that’s “move fast, break things”) or some other big boy with so much success that small fish like us just naturally start repeating whatever they say.

But now that I sit down to explain it, I’m googling for “shipping beats perfection” -khan and not finding much. And while I love its simplicity, “shipping beats perfection” contains subtlety in need of explanation.

We’re willing to be embarrassed about what we haven’t done…

We’re willing to be embarrassed about the things we haven’t done yet. Did you know our mobile app only offers video viewing, not the rest of our interactive platform? Did you know we don’t yet completely cover loops in our programming tutorials and challenges? Did you know we don’t have fully immersive simulations for teaching students physics?

Well, that’s all true, and quite embarrassing…for now. Would we go back in time to delay our mobile app’s launch until we figured out how to support the entire Khan Academy platform in an app? Absolutely not. Would we go back and undo the launch of Khan Academy programming because it doesn’t yet contain all of the content it really needs to? Absolutely not.

Is this the right philosophy for all products? Absolutely not. But educational content is so badly needed right now, and students are so hungry, that it’d be vain of us to think satisfying our own hunger for perfection is worth more than students’ needs. We’ll get to the complete mobile app. We’ll get to better coverage of computer programming content. Maybe we’ll even get to a fully immersive physics simulation. One day.

…but not willing to be embarrassed about what we have done.

“Shipping beats perfection” doesn’t mean we should ship something we’re embarrassed of. Far from it. Would we ever put out physics content that felt crappy or didn’t help students learn? Absolutely not. Would we ever push out a mobile app with a frustrating video viewing experience? Absolutely not.

That would be embarrassment over what we have built. We don’t ship that.

I’m not taking advice from anybody who says ‘shipping beats perfection’…makes no sense anyway you slice it…stability, cache issues, etc.

Concerned Redditor

Concerned Redditor, you need not worry your pretty little karma-filled head. We work hard for performance and stability, and, well, I don’t even know why I’m defending us against this statement because “shipping beats perfection” doesn’t in any way mean “ship crappy code.”

Wait, what? “Shipping beats perfection” can play nice w/ high code quality?

We code review literally every change and demand clear, understandable code. We docstring almost everything. Unit tests are popping up everywhere. If a code reviewer is confused by anything, she can simply say, “I’m having trouble understanding this part,” and it’s on the coder’s head to fix their code or documentation so things are clearer for future readers.

Perfection? Far from it. We can only afford this level of quality and still ship like crazy because we’re willing to be embarrassed about plenty of other things we haven’t gotten to yet.

Bring out the strawman

Try this on for size.

You’re assigned the task of adding a brand new data report for teachers who need to know more about what their students have learned. No doubt in your mind, the report is going to be huge for them. Plus, you just did a design sprint, so you think you know exactly what needs building and how long it should take.

You crack your knuckles and wryly smile as you’re about to fire up your editor and do what you do best.

You crack open javascript/coach-reports/reports.js and can’t believe what you see. What is all this old cruft? Your new report would be soooooo much easier if the data was just bundled up a bit differently. Plus the logic for these other reports is real messy. If you take a couple weeks right now, you could clean things up, and then knocking out your new report will be a breeze a coupl'a days later.

What’s the right move here? Refactor everything and fix it, even though you’ll lose a couple weeks at first? Curse crappy code like a sailor and just hack the new report right on top?

Spoiler alert: we don’t have enough information to answer. How badly do teachers need this? When? Is the “cruft” a bunch of edge cases that really do matter and shouldn’t be thrown away willy-nilly? Is it really old code genuinely in need of replacement?

Every coder faces this demon. The good ones take a step back, ask the above questions, and choose appropriately for each situation. The bad ones dogmatically believe “any code I’m around must be perfect” or, equally as bad, “just ship it.”

Leave it better

You won’t always know the answer, so here’s something for your toolbelt: “leave it better.” Can you liberally add TODOs around the old code, explaining what you will do to fix the situation soon, write your new code such that it demonstrates the new pattern you proudly suggest, and at the same time solve the pressing problem for teachers?.

If so, you left it better. You didn’t delay teachers’ needs for two weeks due to a refactor. You didn’t write lots of new code that you’re embarrassed of. Sure, you may need one messy hack to link your new pattern to the old code. That’s ok, you did so for a reason — for learners — and you added a helpful TODO and a Trello card just to be sure you’ll get back to it. Sure, you’re embarrassed that you didn’t do the full refactor yet. That’s ok, it’s the type of embarrassment — we just haven’t done the work yet — that we’re ok with.

If you’re the type who can’t “just” leave it better but must make code perfect, then you’re satisfying your own needs instead of learners’. You’re violating “shipping beats perfection.”

A story to end on

We’ve seen videos of Spanish-speaking students in South America using Khan Academy to learn math. If a UX guru walked into one of these classrooms in Peru and sat down next to a student, here’s what they’d report back:

  1. Spanish-speaking student goes to www.khanacademy.org.
  2. Student sees a bunch of text in English but clicks around enough to find the math problem she’s trying to practice.
  3. Student selects all of the text in the math problem, then opens another tab, goes to Google translate, pastes the text in, and reads the Spanish translation of the math problem.
  4. Student returns to the tab w/ Khan Academy open, writes her answer, gets the problem right.
  5. Student, seemingly unaware of the usability disaster they’ve just been tortured by, turns to UX guru and smiles blissfully, thrilled by her success.

And then at this point the UX guru’s head would a-splode.

We’ve seen videos of this happening (minus head a-splosion). Many of us have felt deep embarrassment in the past over our lack of translated versions of Khan Academy. But shipping beats perfection. For a long time we weren’t ready to tackle translations. We had to swallow our embarrassment and move forward with the English platform.

So should we be satisfied? Absolutely not. In about a week, a fully internationalized Spanish version of Khan Academy will be out of alpha. Will it be perfect? Far from it. Will 100% of our content translated? Absolutely not. Is our internationalization code free of TODOs and the occasional messy hack? Absolutely not.

Will our internationalization work leave our students, our product, our code quality, and hopefully our world in a better place than before? Absolutely.

You only have to watch one Spanish-speaking student joyfully use an English-only math resource to realize that high code quality and perfect UIs don’t matter for their own sake. They matter when they make a difference for learners. So we leave things better every day, are willing to be embarrassed about what we haven’t done, take pride in what we have, and ship great educational content to everybody as fast as we can.

Shipping beats perfection.