bjk5

Sharing the inspiring, personal stories of Khan Academy users

Tue, 24 Jan 2012 03:50:07 -0500

We have these emails hanging up all over our office, sent in from Khan Academy users with incredible, personal stories to tell. Every time I read a new one I’m emotionally affected, which means my robot emotion chip is faulty.

So when some curious soul (like a reporter) wanders in and asks me, “How will you know if Khan Academy is really successful?” I always answer their (totally valid) question with an explanation of our data, analytics, and fancy metrics — but what I’m really thinking is, “You haven’t read these letters.”

Let’s change that. These students’ and parents’ and teachers’ stories are now available for anyone to be inspired by. It is impossible to read them…go ahead, I challenge you…and not come away with the conclusion that a free educational resource like Khan Academy simply must exist.

Call me a softie. It’s not like I don’t believe in data as the final arbiter of any learning tool’s effectiveness. I do. But if our key data metric happened to be “# of page-long, authentic stories sent in from users who have turned their lives around in the face of drug addiction, unleashed their 2nd-grade son on the advanced math he’s fully qualified to handle, or earned acceptance into a university despite being stuck in a country that does not value education,” I don’t think I’d second guess seeing that number skyrocket.

Hopefully these stories inspire others as much as everybody on our team. We spent time designing the page to celebrate the authors, their letters, and the fact that these are real lives, not product testimonials.

Does the page accomplish this? Feedback welcome, good and bad.

Khan Academy Internship, Fall '11

Sun, 08 Jan 2012 02:49:00 -0500

I already can’t wait to drop some major challenges in the laps of our two incoming Fall interns to see what they can build.

- Khan Academy Internship, Summer ‘11

Check!

David Hu and Julian Pulgarin stepped up to the plate this Fall during their ~~coops~~internships from University of Waterloo. We call ‘em internships because we’re from Amurrrrica, you crazy Canadians.

When hanging with friends and family recently, I found myself shocked by how willing my brain is to completely forget stories that were surely once described as unforgettable. I don’t have any desire to forget what we’re doing at Khan Academy, and that’s kinda sorta why being as open as possible is one of our core dev principles.

The Summer ‘11 story is already a nice piece of shared history that helps me answer every intern candidate who asks, “What kind of project do interns work on at Khan Academy?”. Here’s my version of the Fall story.

If tales about battling a bison for the right to cross a road can get up and walk out of my head,
then…well…better keep blogging.

David

It doesn’t get much more open than David’s post about how we use machine learning to assess student mastery. If I tried to summarize it for you, you’d see my managerial hair start spiking up and up and up toward the ceiling, Pinocchio-style. I won’t insult the work by talking about the statistics. Instead, I’ll just say that we now have a much better understanding of how competent each Khan student is in each math subject. Thanks, David.

That’s not all he did. From a dashboard to emphasize the importance of our exercises to a much smoother way of asking students to review work they’ve done in the past, he covers it all in this Vi Hart-inspired internship post-mortem.

Keeping an eye on our students’ activity via David’s dashboard

Forgetful Ben from the future is grateful for this video. And he’s also super-jacked and smart.

Julian

If anybody reading this has ever beaten Julian Pulgarin in chess, please rub his nose in it. Julian wiped the floor with me so many times that he would beg me to play him while he was blindfolded. My ego can’t handle that sort of hit, so I usually just deleted some data from production and pretended to be disappointed at each newfound emergency’s particularly poor timing.

When he wasn’t humoring me, Julian made major contributions across the board.

He started by building a number of new exercises for students to learn from, including an experimental crack at a new way of teaching fraction intuition. While working on this, he realized that it was painful to test our open source contributor’s GitHub pull requests, so he disappeared for a few days and came back with Sandcastle. Sandcastle automatically tags every pull request with a link that lets our developers test out the requests’ new exercise content in one click.

It’s a shining example of the reason we hire smart people and set them free to get things done. The first time Julian told me about his Sandcastle idea, I didn’t even really understand the direction. Now it’s indispensable.

Julian also gave our first KA Friday Tech Talk about how to do gradual feature rollouts for various segments of a large userbase. He ended up bringing this full-circle at the end of his internship by building Gandalf, our tool for doing the following:

Gandalf lets us selectively roll out features to all sorts of different subsections of our userbase.

“YOU SHA — ok, you guys, Hey!, you guys over there, you can pass — BUT OTHER THAN THEM YOU SHALL NOT PASS.”

Julian even “accidentally” left his chess set in Mountain View for us to mail back to him, which I’m pretty sure was his way of dropping the mic and walking off stage. “You clean up.”

In conclusion, University of Waterloo is legit. I skipped over plenty of work done by both David and Julian, and it still makes for an impressive ~~Fall~~ ~~Winter~~ there are seasons in Mountain View? As argued in the previous internship’s summary, any team that’s not dedicating tons of resources to both recruiting and mentoring interns is plain old missing out. We’re loving every minute of working with our interns.

BONUS!

Much like the phoenix or a tyrannosaurus flying a fighter jet, Summer ‘11 intern Joel Burget has risen from out of nowhere and dropped entertaining stories about his summer’s work. If you read them, are impressed, and want to hire Joel…you can forget it. He’s now full-time Joel.

If you wanna tag in next, the Summer ‘12 internship class still has openings.

.end() makes jQuery DOM traversal beautiful

Mon, 26 Dec 2011 13:52:46 -0500

This is my new favorite jQuery trick. I just learned it this year and have mentioned it in enough code reviews to decide it’s worth sharing.

When manipulating the DOM with jQuery, you often see code that looks something like:

$("#container").show();
$("#container .error").hide();
$("#container .zoo").css("background-color", "white");
$("#container .zoo .monkeys").empty();
$("#container .zoo .title").text("The zoo is empty");
$("#container .zoo input").val("");
$("#container .zoo").animate({height: 250});

…or, if somebody gets concerned about performance, they might try to reduce DOM lookups:

var container = $("#container");
container.show();
container.find(".error").hide();
container.find(".zoo").css("background-color", "white");
...

…and so on. Odds are, when you’re manipulating a single element like container, you’ll probably be doing something to nearby elements in the next few lines of code.

Readers of this blog, meet Emma. Emma, meet the three readers of this blog.

Enter .end(), newlines, indentation, and chaining. When you have one jQuery chain going and modify it with, say, .find(), you’re actually pushing the new chained set of elements onto a stack. .end() pops the current jQuery chain off the stack, which lets you do stuff like this:

$("#container")
    .show()
    .find(".error")
        .hide()
        .end()
    .find(".zoo")
        .css("background-color", "white")
        .find(".monkeys")
            .empty()
            .end()
        .find(".title")
            .text("The zoo is empty")
            .end()
        .find("input")
            .val("")
            .end()
        .animate({height: 250});

It’s easy to read, because the indentation is significant and matches up with DOM nesting levels. It gets rid of unnecessary DOM lookups. But most importantly for me, it feels natural to indent in and out as I write the code, using small additional selectors to step deeper into the DOM and .end() to find my way back out.

I’ve been destroying my old, ugly var’s left and right with this trick. I hope it helps somebody else.

Laughing at Others' Code

Thu, 15 Dec 2011 11:39:00 -0500

Those who have worked with me will know that I’m an expert on this topic because my code gets laughed at all the time.

I’ve seen Good laughing and Bad laughing. Good is what I imagine happens when Robert De Niro sits next to Al Pacino as they’re watching Al’s cameo in some trainwreck of an Adam Sandler movie, and Bob turns to Al smiling and says, “This is awful.” It’s when you stare at some code and think, “Good grief. I can just imagine whatever took priority over making this code more reasonable.” There’s a knowing wink and a friendly jab between the laughed-at ghost-coder and the laugher:

Hah! I almost feel bad that you wound up solving it this way. I’ve been there before. You must be exhausted. Why don’t you sit down, I’ll take it from here.

Good laughter contains respect for the fact that this code exists at all and is being worked on by more than one programmer, which is more than you can say for what I’d bet is a whole boatload of perfectly refactored, unused files littering hard drives around the world right now.

That’s the type of laugher that’s erupting at Khan Academy these days, and it’s no coinkydink that now is right about when a team of top-notch coders are getting their first gazes into some of my previous ~~creations~~abysses. I can say with certainty that in these cases the knowing winks and playful jabs are well-deserved.

The rare (and best) third type of laughter.

A quick story before we get to the Bad laughter. Did you know that when Sal was a one-man show, the entire Khan Academy application was one big main.py file? At least the server-side stuff. All the request handlers, URL mappings, datastore models, data migrations, and even some HTML generation in one big 1000+ line file.

How funny is that! And it stayed that way for months! I mean, please. Who the heck wants to work on a codebase that’s one big file? If you haven’t detected the dripping sarcasm yet, recalibrate your sarcasm detector and start this paragraph again.

It’s easy to see how judging Sal by that one big file while he was busy making 2400+ free educational videos would be like judging a geek for wearing a t-shirt while she was just trying to look presentable enough to get to her computer to start writing code. Guess what? Sal’s code is still around, and it’s responsible for helping teach literally millions of students. Yet, as time goes on, it gets more and more likely that one day somebody will laugh at an out-of-place line with the type of judgment that can only come from being out-of-touch with what really mattered back then…and the fact that that line helped change many students’ lives.

You’ve gotta bend this brilliant tweet a bit to apply it to non-profit education,
but you’ll notice “good code” and “bad code” aren’t on the list.

Bad laughter doesn’t need much explanation, it just lacks respect. It happens too much in our industry, and I’m not sure why. I’m proud to say that it’s not a problem at Khan at all, but we haven’t always been 100% immune. We weren’t always 100% immune at Fog Creek. I’ve been guilty of this laughter myself, and I’d bet money it exists elsewhere. It’s most common among coders who feel like they need to prove themselves, and it can be combatted by a team that emphasizes shipping and the healthy laughter that comes from reminiscing about the last crappy hack you were responsible for when you decided to Just Ship It.

Here’s (just one of) ours:

PLAYLIST_STRUCTURE = [
    {
        "name": "Math",
        "items":
            [
                {
                    "name": "Arithmetic",
                    "playlist": "Arithmetic"
                },
                {
                    "name": "Developmental Math",
                    "items": [
                    ...

That’s our current Khan playlist structure, defined in code. When Sal adds or renames a playlist, we have to change the code. Laugh it up.

This will actually go away soon in favor of something much nicer, and I’m not arguing for crappy code. I’m putting this example here because it got us this far, without many problems, and who knows where we’d be if Sal had spent the first iteration of khanacademy.org trying to decide on the perfect playlist data structure.

…and a more rambling attempt to get a similar point across to a group of UIUC students by playing this powerful Thank You, Khan Academy video before revealing main.py.

After giving logged out users access to pretty much everything*

Sun, 11 Dec 2011 04:48:25 -0500

We had some heated debates a while ago about what would happen if we opened up all of Khan Academy’s content for logged out users. Sal’s videos have always been open in this way, of course, but all the interactive exercises and statistical tracking and badges and stuff required an account.

It felt like the right move when we reconfirmed our belief that educational content should be as open and available as possible. We were also persuaded by Fred Wilson’s belief that giving logged out users more power is an effective way to empower your community. But we worried that registrations would drop, because by handing our 250+ exercises to logged out users, we’d be drastically shrinking the carrot on the other side of the “please login!” boundary.

We decided to go for it this summer. Figured I’d share some results.

A dog

4+ months after the change, we know a little more. Jace looked at the data and found:

Registration rate hasn’t changed. Bit of a surprise for both sides of the debate.
Percentage of our visitors who try an exercise problem increased by (relative) 10-15%.

So more visitors are trying exercises, but they’re not converting into more registered users…yet. We could have fun making up all sorts of explanations. Maybe we need to show off the badges and points unregistered users are accumulating a little bit more, maybe we’re not asking visitors to login forcefully enough, maybe doing math exercises makes people tired, maybe we don’t have enough pictures of golden retrievers on the login page.

Percentage of all visitors who use our exercises

Far more important than any random explanation is the fact that we’re now getting X% more data about how users learn (or don’t learn) thanks to these new exercise visitors. That data is powering some of the best work coming out of Khan Academy so far, so feeding more logged out users into our exercises is, as Jace puts it, a big win on data collection.

Plus, we’re confident that we could iterate and A/B test our way to higher registration rates for our new exercise users. It’s not our priority (for this week, at least). We’re busy cooking up important changes to the core learning experience.

*They don’t have access to everything. The ability to coach and communicate with other users is still restricted to users who have logged in.

How to make "consecutive days of Khan Academy activity" badges

Tue, 29 Nov 2011 22:17:04 -0500

Think how cool it would be if users were rewarded after learning for X consecutive days.
Think about implementation for a second and then go, “Sweet! This’ll be trivial Let’s knock it out Here we go I’ve got things to do!”
Write a line of code or two.
Get sucker-punched by the gruesome specter of timezones when realizing that a student doing work at 8am on Thursday and 9pm on Friday should probably still earn their badge even though over 24 hours of non-work elapsed.
Think “How does Stack Exchange do this?” and discover that while they suffer from the timezone bug, passionate Stack Exchange users have written actual code to do the trick without asking users for timezone information.
Follow those passionate users’ suggestions, knock it out, and enjoy a few weeks later when the most dedicated users start earning badges for 30 days of consecutive learning.

A/B testing still works. [Sarcastic PHEW].

Tue, 15 Nov 2011 02:51:00 -0500

After releasing GAE/Bingo, we received a number of worried correspondences from various very worried correspondents. It seems that GAE/Bingo, along with practically every other A/B testing framework out there, violates some purist principles of how to do significance testing.

The crux of the argument, reworded so simply that I’m pretty sure all statisticians (I admittedly know nothing about stats) would string me up:

If you repeatedly check the results of an experiment, sometimes you’ll see statistically significant results that aren’t actually significant.

So if you’re constantly checking your A/B dashboard and making decisions based on what it tells you, you’re often screwing up.

It’s a mathematically sound argument, as explained to me by my much smarter teammates. And it must be absolutely devastating for all the programmers who went out and bought the Razer Mamba Elite Wireless Gaming Mouse just to increase their click speed so they could mash the refresh button on their A/B dashboards as fast as possible.

Here’s the thing. I know of absolutely nobody who runs an A/B test like a crazed puppy who keeps sprinting loops around your legs hoping that…..ohboyohboyohboyohboy…..after the next 360° you’ll have lowered the puppy treat to the floor.

That doesn’t mean the argument isn’t valid. If you do check your dashboard every 5 seconds like a crazed puppy and immediately end experiments at the first sign of stat sig, then you probably should read the article and…..ummmmm…..find better uses for your time.

Luckily for us, one of my much smarter teammates with much more experience analyzing numbers’n’stuff landed an early modification to GAE/Bingo that should pacify all worried correspondents:

A historical graph of one of our A/B tests and each alternative’s performance. On our dashboard it’s interactive, weeeeeeeee.

By showing this graph everywhere our dashboard shows A/B results and waiting for the results to stabilize, we can be confident that we’re not making a snap judgment in the zone of idiotic decisions.

“Danga zone”

Ok, good. We’re safe. But what about everybody else? Did Fog Creek and 37signals and everybody including Google immediately start hemorrhaging money due to their reliance on faulty A/B tests which this truth came to light???

My guess is no, because A) they aren’t making snap judgments at the first sign of stat sig and B) with significant traffic, many of our A/B test experiments don’t even have a zone of idiotic decisions. Lots of ‘em look something like this:

…and it’s pretty clear in which cases a difference has been made.

App Engine Performance Hacks You'll Probably Never Need, Part I

Sat, 22 Oct 2011 04:08:43 -0400

Building GAE/Bingo required reaching into the bag of performance tricks a couple times. We needed long-term persistence of the data behind many A/B experiments, with stats accumulating at 500 reqs/sec, without slowing down pageloads.

Some of the wabbits we pulled out of the hat are pretty cool. Some are probably really stupid but get the job done. I’ll throw a couple your way and let you choose which is which.

Access a select few from a list of entities on every single request — and you don’t know which until the request is done

GAE/Bingo makes it easy to run A/B experiments in one line of code: ab_test("monkeys"). This means during any given request, an individual user might interact with a number of different experiments depending on various code paths. Without knowing ahead of time which ab_test("monkeys") or ab_test("gorillas") or ab_test("chimpanzees")’s are going to run, we need to minimize the number of roundtrips spent talking to memcache or the datastore.

At first it makes sense to put a collection of all Experiment models in a single memcache slot. This is kinda helpful because at the beginning of the request you can:

experiments = memcache.get("experiments")

…and then each time you need to work with an experiment you can:

experiment = experiments[experiment_name]

You’ve only got one memcache call regardless of how many experiments you’ll be interacting with on each request.

There’s a big problem. Getting a bunch of objects out of memcache also involves deserializing all of those objects, not just transferring them over the wire. And at least until the 2.7 release ships, deserializing objects innnn ppuuurreeee Pyyyyttthhhooonnnnn iiiiiiiiiisssssssssssssss rrrreaaaaaaaaaaaaalllllllllllllyyyyyyyyyyyyy ssssssssssssssssssslllllllllllllllllllllloooooooooooooooooooowwwwwwwwwwwwwwwwwwwwww. If you’re running, like, 50 live A/B experiments and your user needs to interact with, like, three of them when requesting /profile, you’ve got, like, 47 experiment-deserializations worth of performance waste.

Here’s where we pulled out the same trick we first used when building a fast real-time badging system for Khan Academy. Since you’ll almost never be interacting with all of the experiments in an individual request, you really really Really don’t want to spend time deserializing them. So what happens if you serialize the objects once (in this case using protocol buffers) before putting them in the memcache collection?

Things are a little bit slower when creating the collection:

memcache.set(
    "experiments",
    {
        "monkey": db.model_to_protobuf(experiment_monkey).Encode(),
        "gorilla": db.model_to_protobuf(experiment_gorilla).Encode(),
        ...
    }
)

Getting the collection from memcache is MUCH faster:

experiments = memcache.get("experiments")

…because instead of deserializing 50 experiment models, you’re unpacking 50 pre-serialized protocol buffers. Fast. Many much fast.

Now, when you need an actual experiment model, you just:

experiment = db.model_from_protobuf(entity_pb.EntityProto(experiments[experiment_name]))

…and only pay the deserialization penalty when you actually need to use the object.

This screenshot from my recent Khan Academy Friday tech talk should make everything totally clear. Especially without any context, I’m pretty sure seeing a picture of a gnome riding a platypus while carrying a monkey explains everything.

Bottom line: we make one single memcache call coupled with the minimum amount of experiment deserialization necessary, regardless of how many A/B experiments are running or which experiments are used by each request. There are lots of arguments over the fastest way to (de)serialize objects in App Engine — the fastest is to avoid the issue as much as possible.

Where we use this:

Split testing experiments and our real-time badge framework.

put_async without waiting for the put to finish — someone’s gonna yell at me

I’ll get in trouble for this or maybe even ruin the fun for everyone. This involves starting an asynchronous datastore put via db.put_async and then walking away with your hands in your pockets acting like nothing happened.

There are very clear instructions by the talented App Engine team that you should find some time to wait for the get_result() of any call to db.put_async(monkey). It’s also very clear that if you don’t wait for put_async to finish, App Engine is going to wait for you. In other words, you can’t magically send off a bunch of put_async’s and then send your response to the user without waiting for the put to complete.

db.put_async(experiment_monkey)
return render_template("mwuuahaha_im_not_doing_anything_else.html")

You can, however, send off a put_async and then do everything else your request could possibly think of doing (including rendering templates and such) without waiting for the response. App Engine will make sure the response finishes, but if you just kick off the put_async and then walk away and handle the rest of your request, you can maximize concurrency of your request’s work w/ the asynchronous put.

There are *lots* of other ways to get a very similar effect. All of them are probably more kosher. I won’t list them here. This just happens to be a neat little trick that you can trigger with one line of code without worrying about any other boilerplate.

Where we use this:

Find the spots yourself and make fun of me. I have a strong feeling this’ll be replaced in the future.

Long-term persistence without waiting on the datastore

This one doesn’t belong in a “you’ll probably never need” post. It’s extremely common and handy: throw data in memcache (fast) and then run a background task or cron job that persists the data from memcache to the datastore (slow).

We actually get a little trickier because we need to persist lots of data that’s coming in quite often: each and every user’s participation and conversions in each and every A/B test. These events could be triggered multiple times per request for each user. It’s not quite clear how we’d put this data in memcache and what scheme would be running in the background to send it all to the datastore.

We opted for a bucketing system. Every time a user participates or converts in an A/B test, we randomly choose one of 50 memcache buckets and throw their user id in the bucket. When any of those buckets begins to overflow, we fire off a deferred task queue task to poke through the overflowing bucket, pull each user’s data out of memcache, and whisk it into the datastore.

bucket = random.randint(0, 50)
key = "_gae_bingo_identity_bucket:%s" % bucket

list_identities = memcache.get(key) or []
list_identities.append(ident)

if len(list_identities) > 50:
    deferred.defer(persist_gae_bingo_identity_records, list_identities)

Where we use this:

GAE/Bingo and, in simpler fashions, pretty much all over the place.

More wabbits where those came from

If any of these hacks help someone else out, please let me know ASAP so I can win the bet against myself.

Regardless, expect more of these posts in the future. Tricks like these have been critical to keeping Khan Academy fast while adding new tools.

A/Bingo split testing now on App Engine, built for Khan Academy

Tue, 13 Sep 2011 15:32:41 -0400

Continuing my trend of straight-up copying the work of the smartest people I know, I recently decided to tackle Khan Academy’s A/B testing problem (we didn’t have any A/B testing) by bringing Patrick McKenzie’s A/Bingo into App Engine land.

So here you go: GAE/Bingo is released and should get anyone on App Engine up and A/B testing in minutes. It’s currently in production on Khan Academy and performing well with hundreds of requests per second.

Cool! But what’s A/Bingo?

A/Bingo is an A/B testing framework for Ruby on Rails. It’s specifically designed to make the creation of split test experiments as quick and painless as possible.

GAE/Bingo is a re-imagining of this framework’s core design principles inside of Google App Engine. GAE/Bingo was specifically built for use at Khan Academy, which means:

GAE/Bingo is highly optimized for App Engine. Drop it right into any Python App Engine app and you’ll be all set. It’s running in production for us with minimal impact on page load times.
GAE/Bingo works hard to persist experiment data for long periods of time without sacrificing performance.
- Unlike most A/B experiments out there, we may keep an eye on the educational results of an experiment for months and months after a student started down their A/B path.
- Heck, we may even need to correlate our A/B student splits with offline data, like end-of-the-year test scores.

Why do this?

We’ve got a million and one ideas to try out at Khan Academy. What tweak to our game mechanics will best motivate students to challenge themselves? What message makes it most likely for a student to sit back and watch a video when they really need to take time, slow down, and re-learn a core concept?

An A/B testing framework gives us the tools necessary to start answering these questions with experiments and hard(er) data. With ~1.5MM practice exercises answered per school day by Khan Academy students, we have a treasure trove of student activity from which to learn.

We also wanted to spread the love. Patrick helped out the Rails community by open sourcing A/Bingo, and we wanted to do the same for App Engine. I also couldn’t find any Python split testing framework that satisfied our needs and stayed true to the design principles of A/Bingo.

Plus, why not take advantage of the fact that App Engine’s vertical stack empowers framework creators to go pretty far when it comes to creating a drop-in, It Just Works experience? We hope GAE/Bingo accomplishes this and helps out some others in the community.

How’s it work?

Start an A/B test in one line:

# Returns "chimpanzee" to half your users and "zorilla" to the other half
animal = ab_test("cute_logo_animal", ["chimpanzee", "zorilla"])

…and when something good happens, score a conversion in one line:

bingo("cute_logo_animal")

These two lines will automatically take care of experiment creation, user tracking, consistent A/B results for each individual user, and statistical analysis. You can do a lot more, of course, when it comes to specifying alternatives and tracking conversions — check out the docs. There are some pretty simple (optional) hooks that make it very easy to get consistent A/B results even when your users transition from anonymous to logged-in.

Trivial example: an A/B test proving that messaging a student with “You’re ready to move on!” is statistically more likely to encourage a student to move into to new content than “Nice work!”

Once at least one user causes the above lines of code to execute, you’ll get statistical analysis and be able to control your experiment from the dashboard.

Can I use it? Can I tweak it?

Please do. Let me know how it goes via Twitter or email. Patch up all the inevitable bugs and fill in all the major holes left by our desire to ship v1. We’ll be continuing to improve the framework, and all help is welcome.

Enormous thanks goes out to Patrick McKenzie for his framework’s inspiration and the encouragement to follow his lead. I’ll be blogging more in the future about how we’re using GAE/Bingo and how we keep track of hundreds of requests per second w/ persistent storage and minimal impact on page load times.

Khan Academy Internship, Summer '11

Fri, 12 Aug 2011 12:47:00 -0400

Want to be handed a major portion of Khan Academy ownership, ridiculously high expectations, and a bunch of mentorship from our full-time devs? Sign up now. We believe anybody can help the world get a great education, and we accept interns year-round.

I can’t remember a time in my history of small company software development that hasn’t felt like sitting in a rickety donkey kong cart with jet afterburners attached and blazing, and everybody inside is just trying to keep the staples (why did they use staples?) to hold long enough for us to make it around the next bend in the tracks.

That being said, at least in my limited donkey kong cart experience, summer internship seasons always stand out in my mind as new high watermarks for shipping speed and dev intensity.

Just like Sal’s videos, our practice exercises are at the very heart of Khan Academy.

This summer’s class of Khan Academy interns has been no different. Our interns come in with promises of being handed ownership and control over major, user-facing features, and in return we demand excellence; it’s pretty similar, actually, to Khan Academy’s educational belief of encouraging experimentation but expecting mastery. They were shipping features on day one.

Any dev team out there not acknowledging the fact that high school and college students are capable of showing up on your doorstep and almost immediately redefining major portions of your product for the better is either failing to recruit well or is plain old missing out. Big time.

If that’s you, I hope to change your mind with this post.

Each of the following improvements to Khan Academy was contributed either largely or entirely by our interns this summer. Four of ‘em are in college, one just graduated high school, and one hasn’t even started applying to colleges yet.

An entirely new collection of practice exercises built on an entirely new exercise framework.

This is a major body of work.

We learned about all types of weaknesses in our old exercises after last year’s pilots, and we’ve tackled them head on by improving our hints, removing multiple choice answers, focusing on the user’s exercise experience, and building new ways of asking old questions.

Almost all multiple choice questions are gone.

We’ve focused on helping our developers and our community create new exercises quickly. We’ve written better documentation, shipped simpler dev tools, and built solid bug reporting workflows to maintain a healthy stream of new, quality exercises. Our interns are responsible for not only porting all existing exercises to our new tools but also developing brand new frameworks to help exercises draw graphs, randomize questions, generate procedural hints, and more.

The Summer 11 interns’ recently launched new exercises have already served up over 5,000,000 math problems to Khan Academy students.

We know that at the end of the day, the only thing that matters is whether or not Khan Academy students are really learning, and a large quantity of quality interactive exercise content is core to that mission. KA wouldn’t be what it is today without the large quantity of quality videos Sal has created. As a development team and community, we should consider ourselves challenged to match his videos with quality exercises. The team’s efforts this summer have given us the tools necessary to take a crack at this considerable challenge.

In fact, exercises are so important to us that we’re now hiring full-time exercise developers to come push these tools to their limits and redefine what it means to learn online. If you want to join some very passionate devs, either apply now or join our open source exercise community.

Full access for unregistered users

Followers of this blog (also: leprechauns, unicorns) may remember a post in which I struggled to decide if Khan Academy should follow Stack Overflow’s registration model by allowing non-logged-in users to participate in all of our content and automatically transferring their work to a permanent account after they log in.

The more we thought about it and read posts about unregistered users like Fred Wilson’s, the more we realized that making access to our educational content as easy as possible was the right thing to do.

We track users’ progress and encourage them to log in, but we never get in anyone’s way when they’re trying to learn.

Two interns completely owned the design and implementation of this feature. If you go to Khan Academy now, you can start earning points and badges for watching videos and working on exercises without ever logging in. We encourage users to login at various milestones in their progress, but we never stop them from continuing to use the site or force them to close a popup.

If you log in, you keep all your progress. This has significantly reduced our bounce rate by getting rid of painful login walls, and we’re continuing to watch other statistics to see the effects of this change.

Video progress tracking

I can’t count of the number of times I’ve heard users ask us to display individual video progress next to each one of the 2,500+ video links on our homepage. I’m not gonna get into the various technical challenges here (maybe a different blog post), but this is nontrivial for a brain like mine.

Luckily, we hire interns that are way smarter than me and are able to solve such problems. These days, whenever you watch a piece of a Khan Academy video, skip around it, pause it, play it, or whatever, we keep track of extremely precise video progress statistics and display useful progress indicators next to every video link.

Twitter and Facebook sharing

One of the earliest features launched during Summer ‘11 was the ability to share videos, exercises, and badges on Facebook and Twitter.

Facebook sharing has slowly gone up (even during the natural academic lull of summer), and we hope this trend will continue.

All that and more, powered by interns

…and those are just the bigger changes. Our interns continue to claw for inch after inch after inch of improvements like nicer internal statistics, faster deploy scripts, performance tweaks, and better user account management.

Summer’s not over yet. Omar Rizwan, Jeff Ruberg, Joel Burget, Igor Terzic, Parker Kuivila, and Ben Alpert continue to set the current bar for Khan Academy internship classes. As an organization, we aim to beat this mark in the future. But those six are pushing hard to make it a tough task, and khanacademy.org is improving for the better, quickly.

Don’t doubt the inexperienced. Get your team’s recruiting, mentorship, and code reviews right (easier said than done), and a summer internship can be one of the best things that’s ever happened to your product. I already can’t wait to drop some major challenges in the laps of our two incoming Fall interns to see what they can build.

Join us

Big shoes to fill.

Fast and slow queues on App Engine

Tue, 19 Jul 2011 03:59:00 -0400

tl;dr — If you’re using task queues on App Engine and your task execution speeds vary greatly, you can get yourself into serious performance trouble. We addressed this by explicitly separating fast and slow tasks, and we released a little utility to help you do the same.

Only one user has ever earned this badge

Quick story: even though we put in a lot of work making sure Khan Academy users are awarded badges in real-time, we also run background badging processes to make sure we didn’t miss any. This process uses GAE’s mapreduce framework to map over all users and make sure their badges are up-to-date, and it looks something like:

# Called once for every user
def badge_background_check(user):
    if user.has_recent_activity():
        user.update_badges()

has_recent_activity is really fast, and update_badges can be really slow. This doesn’t play nice with App Engine’s request scheduler, and here’s why.

When you start firing off a bunch of tasks to a specific task queue, App Engine keeps a running average of how long tasks in that particular queue are taking. If they’re really fast (< 1000ms is what we’ve heard…), you can think of the task queue being painted with a shiny yellow smiley face. If they’re slow, the queue gets a sad red frowny face stamped on its forehead.

Don’t judge just because the sad one sounds like Eeyore. Neither of these situations cause problems.

If you’ve got a shiny yellow smiley face, App Engine’s scheduler will try to get your tasks done quickly by queuing up your fast tasks in the same lines that your users’ requests wait in. As long as these tasks stay fast, your app will scale when more instances are needed, your users should never notice that they’re standing in the same line as the fast tasks, and all is good.

Thing is, it’s even ok to have a frowny face. App Engine will use a different, slower scheduler to hand out your work without interrupting user facing requests. You’ll never wait in the same line as a user request, and you can take all the time in the world (as long as the world only lasts for the next 10 minutes) without worrying about someone emailing and complaining that your site’s gotten really slow lately.

The misery starts when you have a queue that’s mostly fast but hits sporadically slow tasks. Now your queue is all shiny and yellow and smiley and sharing the same checkout lines as user facing requests, but every once in a while a grumpy red frowny face stands in line, takes five full minutes at the cash register asking about the store’s raincheck coupon policy, and ruins everybody’s day. Your queue’s sporadic behavior is now directly hurting your users’ perceived performance.

This has caused us some very serious perf hiccups. 90% of our users don’t need to run update_badges on any given day, so the average task speed is very fast. When we hit a slow task, instance request queues can grind to a halt. At our worst, we’ve seen user facing requests sit waiting in the request queue for 9000ms (that’s approximately one full eternity) before our code even gets a chance to run.*

I was going to put an animated gif of a yellow smiley face that suddenly flashes a red frowny face here, but then I remembered that I’d have to kill myself for having an animated gif.

The gae_fast_slow_queue utility helps work around this as quickly as possible by making it easy to run a bunch of fast tasks while identifying slow tasks and splitting them off into a separate queue:

@fast_slow_queue.handler(lambda user: user.has_recent_activity())
def badge_background_check(user):
    user.update_badges()

Now, when badge_background_check gets queued up, it’ll always only run the fast lambda function to keep the main queue yellow and smiley. If real work needs to be done, it fires off a different task in a different queue, and this one is guaranteed by our utility to take at least 1000ms so it stays red and frowny forever. GAE’s scheduler will steer these clear of user facing requests, and you don’t have to worry about your task impacting users’ perceived perf.

* We noticed this after deploying gae_mini_profiler and browsing around the site — every once in a while a developer got frustrated waiting forever for a page to load, then the profiler would come back and add insult to injury by screaming, “Woohoo this page only took 150ms!” You can confirm issues like these by looking for high pending_ms values in your logs.

App Engine Mini Profiler and Temporary Redirects

Fri, 08 Jul 2011 01:23:00 -0400

We just made a small change to Google App Engine Mini Profiler to deal with the fact that spitting out profiling stats on every rendered page still misses all the pages that aren’t rendered: the POST side of temporary redirects.

It’s super common to submit a POST and redirect to a GET. When using a profiler that spits out performance data about the currently rendered page, you lose all of that profiling data about the POST request, even though the user still had to wait for the request to finish.

Not no more

Sticking with the belief that making the path from developers to profiling data as short as possible will produce a faster site, we now automatically expose these redirects and all of their performance stats. App Engine Mini Profiler handles this with ajax requests as well (you’ll see two profiler entries in the corner when sending an ajax POST and getting redirected to a GET full of JSON data), and in the unfortunate event that you’re bouncing the user around a long chain of redirects (more than one and you should feel guilty), it’ll keep track of all ‘em.

Exposing the redirect’s data

Google App Engine Mini Profiler

Sun, 26 Jun 2011 14:39:00 -0400

Patrick Bateman: (Looking at the business card) “Look at that subtle off-white coloring. The tasteful thickness of it. Oh my god…”

I finally understood the full extent of Bateman’s jealousy recently when reading about the MVC Mini Profiler created by the Stack Exchange team. My eyes scanned all the features, and I started to lose my mind:

“…live, instant feedback on the performance characteristics of the pages we are visiting – in production,”
“…ubiquitous,”
“…only visible to developers and has no noticeable impact on the site’s performance,”
“…drill-down on and share profile sessions,”
“…ajax calls are displayed as they happen,”
“…duplicate query detection,”
“…drop in module for any MVC project,”

…my god, it even had a watermark.

As Jeff said, simply showing a render time for all pages can be critical

It really hit home for me when reading Sam Saffron’s post: “No web frameworks seem to provide a comprehensive approach to page profiling out of the box.” It’s true. This level of ubiquitous profiling baked right into all major web frameworks would really change the game. I decided I couldn’t live without something like mvc-mini-profiler any longer, so we just released gae_mini_profiler.

gae_mini_profiler is a drop-in, ubiquitous, production profiling tool for Google App Engine heavily inspired by MVC Mini Profiler. Since App Engine and MVC have so many fundamental differences, the tools aren’t identical, and it’s not really a port as much as a ridiculously kindred spirit. If you want, you can play around with a gae_mini_profiler-enabled demo of GAE’s example chess app — in this demo case, the profiler is enabled for all users.

Taking a glance at one of our slower pages in production

I’m confident that always having all of this data right in front of our developers as we browse around the live Khan Academy site will improve performance. I’ve already noticed a few problems I didn’t previously know about in just the first few hours after deploying.

AJAX calls stack up as they come in, ready for examination

You can dive deep on the details of each request

All props go to Jarrod, Sam, and probably the rest of the Stack Exchange team for blazing the path with mvc-mini-profiler and inspiring this tool. I borrowed ideas ranging from basic UI to dupe query detection to jQuery.tmpl usage to ajax request stacking to lots of other stuff I can’t remember. The sizable differences in the tools really spawn from the type of data I get from Appstats and cProfile.

The project is obviously young, but any App Engine app should be able to drop it in quite quickly with minimal configuration. Instructions are at the repo, and I hope others find it useful.

“…relief washes over me in an awesome wave.”

Connecting the students

Sat, 18 Jun 2011 03:46:00 -0400

We’re thinking about how to connect our community of students, and there’s a lot we don’t know. Here’s the little we do:

Students have watched Sal’s videos over 60,000,000 times and have answered 75,000,000 practice problems.
This is a massive amount of learning, but much of it has been done by students who are relatively disconnected from their peers.
Nothing indicates true mastery of a subject like the ability to teach it to others.
The small (relative to our total userbase) number of students using Khan Academy alongside their peers in a school setting engage the software in incredible ways. They work together in groups, challenge each other, and cheer for their friends when they earn badges. Working with others, especially a fearless teacher or coach, amplifies Khan Academy’s effect significantly.

We want to build these amplifiers directly into Khan Academy. No matter which direction we brainstorm, the possibilities for connecting our users don’t seem to stop. A small subset of what we’ve considered:

Any struggling student could ask for immediate help from a qualified peer or pre-approved teacher, who could immediately see the student’s past work and mistakes.
Any teacher, coach, or student (the lines should be blurred) could add their own content in the form of both videos and exercises to supplement the KA library according to their own needs while taking advantage of our platform, dashboard, and analytics.
Users could participate in message board-like communities about all types of KA-related content.
Students could team up to work through KA content together while facing all kinds of cooperative challenges.
…doesn’t stop there.

While we don’t claim to have the answers, you’ll see more and more experiments coming from our core belief that teaching others is a great way to learn. We’ve already dabbled (can’t emphasize that enough…^dabbled) with this goal in the question and answer section beneath each of our videos:

Students teaching students

It seems likely that Sal’s videos each have a few common misunderstandings or logical jumps that trick the majority of viewers, and this feature is designed to let the community fill in the most important gaps.

Even little questions like these are important

Simple voting mechanics (with one or two doodads from the best voting implementation out there) hopefully surface the most common questions

This small experiment has already taught us some important lessons. I see no reason to stop the lazy list writing style I started above:

There’s a big difference between fostering a community that’s focused on creating and refining artifacts (think StackOverflow and Wikipedia) and a community that encourages any and all mentoring between students. In a community that creates artifacts, duplicate content is the enemy and efficiently finding others’ answers is the holy grail. For us, there’s something very important about a kid raising her hand, asking why subtracting a negative number moves you to the right on the number line, and working towards the answer with another human being (or human being-like coach). It doesn’t matter how many people have asked the question previously.
Sep Kamvar gave us some great advice along those lines the other day when he reminded us that the most efficient system (think about the ability to instantly find all others’ questions and answers) is not necessarily the best solution when encouraging interaction. I’ll put myself in the hot seat and admit that as a developer, that takes some getting used to.
Our content is frequented by different age groups, which creates different community dynamics for each subject.
As expected, even this small feature generates a wide range of content quality. Not all participation is helpful, and our system isn’t perfect.
It’s true that each video comes with some common curiosities, and a few members of our community are fantastic at responding to them.

We’ll be experimenting more, and we’re always listening. The direction of educational communities online feels wide open, and every day we have to pick our path. Share your opinions, they matter a lot.

Opening up the Khan Academy API

Sun, 05 Jun 2011 14:06:00 -0400

As soon as I joined KA, I began receiving emails from people asking to integrate with Sal’s content in various ways: in mobile apps, in search engine results, in other educational sites. So we threw together the quickest, ugliest API we could get away with at the time in order to give other developers access to our playlist and video library.

It’s now a few months later, and that hack of an API no longer cuts it. It lacks authentication, user-specific data, API versioning, and other basics. We need these basics for the official mobile apps we’re developing, and the development community needs them to make apps that do more than link to our videos. Which is why we just released what we’re calling API version 1.0.

The new API gives developers access to not just playlists and videos, but also exercises, badges, users, and logs of historical data about each and every problem or video students have watched. I’m sure we missed a lot of stuff, so developers: just let us know when you need access to something else.

Using the API to check out my performance in our Writing Expressions 1 exercise.

We’re slotted to be one of the featured APIs in next weekend’s hack4knowledge — I’m crossing my fingers we see some cool educational hacks using our data.

Pickles and memcache

Mon, 23 May 2011 03:52:50 -0400

My recent time spent creating a hack that I’d later delete taught me a bunch about Python and Django. One cool little trick I wound up using was a dictionary that automatically pickles and unpickles its contents.

I needed memcache to quickly deserialize a dictionary of values, and it wasn’t always guaranteed that I would use all of the objects in the dictionary with each request. Sometimes the contents of the dictionary took a long time to deserialize (lists of App Engine models), so it felt really wasteful waiting for pickle.loads to finish its stuff when we didn’t need half the results.

The result was a dictionary that pickles its own values whenever items are set and depickles them automatically on get.

import pickle

# We use a context dict full of pre-pickled values for our cached two_pass_template context,
# so we don't waste time de-pickling entities that aren't used in the second pass.

class PickleContextVal:
    def __init__(self, pickled, val):
        self.pickled = pickled
        self.value = pickle.dumps(val) if self.pickled else val

    def val(self):
        if self.pickled:
            self.value = pickle.loads(self.value)
            self.pickled = False
            return self.value
        else:
            return self.value

# See http://stackoverflow.com/questions/2060972/subclassing-python-dictionary-to-override-setitem
class PickleContextDict(dict):

    def __getitem__(self, key):
        pickle_context_val = super(PickleContextDict, self).__getitem__(key)
        return pickle_context_val.val()

    def __setitem__(self, key, value, pickled=True):
        super(PickleContextDict, self).__setitem__(key, PickleContextVal(pickled, value))

Depickling strings is really really fast, so by spending a tiny bit of extra time pickling all of the individual items of the dictionary and then pickling the entire dictionary once more before storing it in memcache, we spend very little time deserializing objects that weren’t needed.

How much I enjoy deleting my own code

Sat, 21 May 2011 11:29:56 -0400

I’m not sure why a few months makes such a difference in how I feel about removing my own work. Maybe there’s some intuition that code hasn’t gotten to serve its real purpose without being out in the wild for a couple months. I don’t know. I do know that after a few months, when I’m free from the worry that I’ve recently wasted my time, nothing feels better than removing (not rewriting!) your own code.

I know this because as soon as I post this blog I’m going to delete what I consider to be a cool piece of code I wrote last week. It took me a while to work past the “this hack is sweet it’s gotta be useful” mental roadblock and listen to my inner coder’s screams that the payoff is just not worth the hack’s future headaches.

I wanted to speed up our video pages (we’re perf junkies) and in doing so, solve the following problem. Usually, you think of cache strategies like this:

…but for pages with small amounts of user-specific data, it makes sense to think of caching like this:

One way people approach this is to use two-phase templates. Your first pass fills in the cacheable stuff, leaving the raw template unrendered in some spots. You keep this incomplete template around for all future requests, which just run the quick second pass to render the dynamic stuff. All solutions I’ve seen for this require messy template syntax and lots of work to set up the whole rigmarole. I wanted a way to do this without requiring knowledge of the special non-cacheable bits when writing your Django templates, so we’d be sure to use it over and over again just by adding a Python decorator here and there.

The goal was to write a normal template like this:


    
        {% if user %}
            Logged in: {{ user.name }} - {{ user.points }}
        {% endif %}
        {% video_content lots_of_datastore_data %}

…and call it like this:

render_template("template.html", 
    {"lots_of_datastore_data": video_data}
)

…and have it automatically recognize which template variables were missing (notice there’s no user in the template context dictionary above), to produce this:


    
        {% if user %}
            Logged in: {{ user.name }} - {{ user.points }}
        {% endif %}
        
            ...

…while handling all the weird template situations that you can expect in the real world when not writing overly simplistic blog examples.

Anyway, that’s what the code does. And it automatically caches the first render pass of the template, and it only runs the second part of the request handler that produces user-specific data for every subsequent request so you don’t have to repeatedly load lots_of_datastore_data, and it uses memcache to be really super fast, and you barely have to add more than some @two_pass_template decorators to get it all to work, and and and and and.

What’s the problem?

To seamlessly handle all the missing variable magic without bothering template writers, I had to create deeper monkey patches in the Django internals than I feel comfortable maintaining. This level of monkeying with django’s internals can’t possibly be done totally correctly (by me) w/out long-term debugging fiascos down the road. If someone else were maintaining this, I’d be far more likely to get involved.
This type of caching is really useful in rare circumstances. It’s great for our video page, but I’m having trouble finding other applications.
We face maintenance nightmares whenever upgrading Django (we’re still on 0.96) or, obviously, using a different template system).

I have an extremely high tolerance for pragmatic hacks, especially when dealing w/ headaches as a developer just to grab 10 or 20 milliseconds of perf for our users (that’s pretty much the definition of client-side performance work). This particular hack cut our video page’s 100-200ms responses in half and protected us from occasional datastore latency spikes! However, we can get the same benefits for larger parts of our applications and with much less technical debt via normal caching techniques and pursuing our switch to the HR datastore, respectively.

So as soon as I click post I’m removing the code, even though it’s still warm. It’s in source control forever and ever, waiting for when we’ve taken care of the bigger’n’better performance wins and are still hunting for milliseconds years from now. Or waiting for somebody else to make the magic much more maintainable.

Raising expectations for educational tech speed

Sun, 08 May 2011 20:13:30 -0400

Khan Academy and the rising ecosystem of edutech startups aren’t going to come out of the gate with any perfect educational solutions (or anything close). We can raise the bar by building teams and products that move quickly in response to student data and the needs of learners. And teachers can, eventually, start to expect this well-deserved treatment instead of reeling in shock when their needs are addressed.

I’ve heard the motto “Get S*** Done” happily muttered more and more from our dev team recently (not just from me). The mindset of moving fast with small, steady improvements every single day is one gift it feels inarguably right to bring from the world of ~~agile~~ ~~lean~~ quick startups into that of educational tools.

A few examples of the type of speed I’m talking about, all courtesy of the talented missmarcialee.com:

Our prime factorization exercise offered multiple choice answers to students in a way that made it easy to game the correct answer without understanding prime factorization. Teachers and students recognized this, told us, and Marcia fixed it, for the entire world, within hours (she removed multiple choice).
For the minority of exercises that still have multiple choice, we heard that students were sometimes able to spot various hints in the offered choices that made it easier to deduce correct answers. We immediately added more frequent “None of the Above” options to keep the challenge honest.
As soon as “None of the Above” popped up more frequently, Marcia heard that whenever it was selected as the correct choice, students wanted to see the actual answer so they could verify their work. The next morning, whenever “None of the Above” was correctly selected, it slowly faded out and replaced itself with the actual answer.

Three pieces of feedback with three tiny changes that add up. Soon we’ll be driving these increments with more and more of our hard data (about 15 practice problems are answered per second during peak right now).

Everyone’s still much closer to this than where we’re heading.

Saying that Khan Academy is in the infancy of what educational technology will become doesn’t always hit home for students, teachers, whoever. But when they experience this rapid iteration in response to their feedback, you can see in their eyes and read in their tweets that this isn’t the type of interaction they’re used to with their educational tools. A few years of this and KA, along with its contemporary tools that are popping up, will look nothing like it did before.

More importantly, a few years of this and teachers and students will expect more of their tools and the team behinds them.

That’s why it’s exciting to read about educational startups and funds popping up. FWIW, we’ll keep doing our part in making this level of responsiveness become the newly expected norm in educational content.

P.S. I put my money where my mouth is and just moved to the bay area this weekend to join the team in-person. The closer we are to our pilot programs, the faster we can iterate with students and teachers.

Profiling client-side performance over time

Mon, 25 Apr 2011 23:54:12 -0400

Steve Souders leads the way when it comes to client-side performance. A few years ago, his High Performance Web Sites book drilled the following into my head:

Most devs spend time optimizing server-side performance, but your users probably spend way more time waiting around for all types of client-side junk.

That book paved the way for great client-side perf tools like YSlow and, not long after, Page Speed. This post is about using another service, GTmetrix, to monitor changes in your site’s YSlow and Page Speed scores over time.

This khanacademy.org request timeline is fairly normal for most pages out there. The server’s entire response has been received by the browser in 227ms, but the DOM and body load events (blue and red lines, respectively) are way out to the right. Even if we halved our server’s response time, we’d only make a tiny dent in perceived performance.

The trouble with client-side perf is it’s really sneaky. It’s not nearly as straightforward as avoiding O(n²) algorithms in your server-side code. You have to think about request counts, request sizes, minification, DNS lookups, cache rules, proxy caching, asynchronous downloads, redirects, unused javascript, and more.

It’s way more difficult to keep all of these rules in mind when adding a new feature to your site than it is to avoid slow server code. You’re making a new popup dialog for your site? You add an icon here, some

As always, you’re immediately faced with a decision about how to version this file. You always want visitors to use a cached version of this file unless something has changed. If you leave the URL as-is above, you’ll be telling users to “just press Ctrl+R” a lot after you deploy a new version and their cached, stale javascript no longer works. A common, easy solution is to append the current application’s version to the URL via query string, like this:

…and now, whenever you deploy a new version of your app, every browser that hits your site will be guaranteed to download the latest and greatest version of your javascript. Up until today, this is the technique we used for Khan Academy.

There are three problems here.

If you deploy frequently (we do), you’re making all your users re-download all your resources all the time. If you’re deploying three times a day every day, you might as well disable caching entirely. For the past few months, every morning when students in the Los Altos School District’s pilot program logged on to KA, they’d be downloading a new script even if it was identical to the cached version already sitting in their browser, begging to be used.
If Google’s Page Speed is to be trusted (I choose to believe), query strings cause some proxies to skip caching your static resource, no matter what. This means any query string-based cache busting scheme you’ve got in place from ?v=3 to ?hash=AB34 will ruin some perfectly good proxy caching opportunities.
…and last, but not least, if you’re on App Engine and have a nontrivial amount of traffic, this form of cache busting is downright dangerous. Read on.

App Engine’s "all.js?v={{Version}}" dilemma
The following is complete conjecture that is sure to be mocked by somebody with real knowledge of the situation.

When you deploy a new version on App Engine, the 5, 10, 100 different instances running your app don’t switch over to the new version all at once. They switch piecemeal.

Consider the following scenario:

You deploy a new version, V5, which is propagating across your servers. Great. Almost past “???” and into “Profit.”
A loyal user visits your site and gets routed to a server just itching to show off its brand new version.
New version of page sent to user. Smooth sailing.
User’s browser checks out the new HTML and notices a reference to the never-before-seen all.js?v=5.
The browser dutifully asks Google’s servers for the new version. App Engine’s static resource CDN has never seen this file before, so it has to go fetch the file from somewhere.
Since your app isn’t fully done deploying and you’re just not that lucky, the server that responds to the CDN’s request still has an old version of all.js.
And here’s where you get hosed. Sure, your app is fully deployed, but the CDN holds onto that old version and continues to serve it. No amount of Ctrl+R’ing can save you now. All requests for all.js?v=5 will continue to serve v4 contents for quite a long time, and the browser is powerless to avoid it.

This bug started to bite us during what seemed like every other deploy after we crossed a certain traffic threshold (and therefore increased our race condition odds). The worst part is that geographically separate CDNs mean everything can seem fine on the east coast while the entire west coast is unable to use Khan Academy thanks to old javascript files.

All three of the above problems are solved by altering your javascript package’s filename with a hash of the package’s actual content. Now, our javascript references look like this:

…which means we never ask users to download scripts they already have, we take advantage of proxy caching, and we protect ourselves from bugs caused by partial deploys and zealous CDN caching.

bjk5

Sharing the inspiring, personal stories of Khan Academy users

Khan Academy Internship, Fall '11

David

Julian

BONUS!

.end() makes jQuery DOM traversal beautiful

Laughing at Others' Code

After giving logged out users access to pretty much everything*

How to make "consecutive days of Khan Academy activity" badges

A/B testing still works. [Sarcastic *PHEW*].

App Engine Performance Hacks You'll Probably Never Need, Part I

Access a select few from a list of entities on every single request — and you don’t know which until the request is done

put_async without waiting for the put to finish — someone’s gonna yell at me

Long-term persistence without waiting on the datastore

More wabbits where those came from

A/Bingo split testing now on App Engine, built for Khan Academy

Cool! But what’s A/Bingo?

Why do this?

How’s it work?

Can I use it? Can I tweak it?

Khan Academy Internship, Summer '11

An entirely new collection of practice exercises built on an entirely new exercise framework.

Full access for unregistered users

Video progress tracking

Twitter and Facebook sharing

All that and more, powered by interns

Join us

Fast and slow queues on App Engine

App Engine Mini Profiler and Temporary Redirects

Google App Engine Mini Profiler

Connecting the students

Opening up the Khan Academy API

Pickles and memcache

How much I enjoy deleting my own code

Raising expectations for educational tech speed

Profiling client-side performance over time

A/B testing still works. [Sarcastic PHEW].