Ben Kamens

You must stop coding, you must keep coding

Sun, 19 May 2013 13:31:00 -0700

Our dev team’s grown to twenty-three and when the summer intern zenith hits we’ll be — this is weird to type — forty for a few frightening days.

My biggest personal challenge now, by far: How do you stay connected to a product amidst all this scary m͓͇̘̗a̴̹̠̜̝͖̘n̩͙̘͙̫ͅa̬͙̖̬͍̦͔g͓̼͈͇̙i̪̺̘̙̻͘n͕̠̰̦̗g̗? Do you launch the damn editor? Write a single line of code?

I’ve been trying to squeeze every last bit of advice out of role-models I respect. I’m seeking stories from folks who’ve successfully navigated these waters. From personally writing tons of code in v1 to scaling and managing teams who then receive profuse apologies for all those v1 hacks. If this is you, plz send tips.

Here’re the various mentor voices in my head so far.

You must stop coding. But you must not stop coding. Stay very connected to your product. Make mockups and decisions and docs. Scale your team. Intimately know your tech and your tools. Don’t get out-of-date. Be a coder, but don’t waste time in the code.

You absolutely without a doubt need to keep coding, but under no circumstances should you be coding.

Got it. ✔ Chhhheck!

As confusing as that seems, there is solid agreement for how a technical manager should spend, say, 85% of her time. Scaling, recruiting, empowering, refining some teams’ focus, blah blah blah BLAH we’ve all heard it. Everybody agrees.

It’s the other 15% of the day that’s up for debate. This is that short bit of time when you do something creative that magically keeps you tied to your product and on top of your tech stack and immunized against pointy-haired-boss disease. Some suggest mockups, some suggest small production bug fixes, some suggest adding unit tests, some suggest writing code that you never intend to check in.

Being a coder without coding

There’s leeway here, so for anybody else who’s trying to figure it out, here’s how I’ve chosen to spend my intentionally non-managerial time.

Fixing small bugs for various teams. From a performance tweak to fixing a x-browser bug. Small, maybe 1 hour of work. I see the code, I make sure my dev tools are up-to-date, I ship, I communicate what I fixed. In and out. I can lightly connect with multiple teams this way.
Pairing with somebody for 1 or 2 hours a week. I work with a developer who’s extremely effective at making use of my time if I give it to him. I put a block on our calendars, show up at his desk, and do exactly what he tells me to do. His project moves a bit faster, he steals whatever codebase knowledge I have left from earlier days, and I get to make a difference in our product by following his lead. I don’t personally think this would work if I showed up at just anybody’s desk. He’s unique, I’m lucky.
Asking teams to send 1-4 hour dev tasks my way. I’d normally assume that this would overload me and I’d have to turn everybody down, but I’m actually still waiting on these tasks. *wink*.
Participating in hackathons. I am excited to run another healthy hackathon this summer just so I can drop my managerial hat, hide in a corner, and hack.
Using the weekends. This is a dirty secret. Little management needs to happen on weekends, so I use some free time to hack. Controversial and not a great tip to give everyone, but it’s important to me.
Shutting myself away for a few days and coding a small feature. Last time I did this? Months ago. Last time I did this without feeling that I’d shirked more important duties? Over a year ago. I no longer think it’s wise. “You must stop coding.”

As confused as me?

Rands’ Technicality is the authoritative piece on this search for maker/manager balance. He opens with “Stop coding.” By the end he commands, “Don’t stop developing.”

What’s clear is these waters are murky. Any good technical manager must go through the difficult transition from coding to spending almost their entire day empowering smarter teammates. But how you maintain that certain je ne sais quoi that separates you from the non-technical manager you never want to be is a pretty personal choice.

Me? I have to launch the damn editor. Just for 30 minutes. Not to code, but to code a single line.

The Absolute Minimum Every Khan Academy Developer Absolutely, Positively Must Know About Mentoring Interns

Fri, 22 Mar 2013 02:13:00 -0700

Heard a bit of an interview with Dick Costolo while driving home. He was telling the story of why he started his “Managing at Twitter” course that he personally teaches to all new managers. It went something like so:

An engineer came up to me who had recently switched from one Twitter dev team to another.

He said, “My old manager used to hold 1:1 meetings with me. My new one doesn’t believe in them. Which is it? Should I be having 1:1 meetings with my manager at Twitter?”

Supposedly this kicked off a wave of realizations in Dick’s head about all the inconsistencies in Twitter’s management styles. He wanted to fix the problem, started his “Managing at Twitter” course, and you’d have to ask some manager at Twitter what happened next because how the hell would I know if it’s working.

Working on v1.1 of “Managing at Twitter” that I teach to managers at Twitter. - thought it would end up being a 2hr class & it’s 6 hours.

November 12, 2011

The story did strike a chord with me. As Khan Academy’s pool of new mentors grows while interns simultaneously flood in the door (13 confirmed so far this summer, join up), it feels healthy to head off any mentorship inconsistencies at the pass.

I want simple and uncharacteristically terse guidelines for Khan Academy mentors. Here goes.

Explicitly say “I’m your mentor, I’m here to help” on the first day.
Don’t have to use those words, but you get the picture. Leave no doubt about what your intern should do if they’re stuck.
Hold weekly 1:1s.
A 12-14 week internship flies by. These 1:1s are our best checkpoints for getting feedback, giving mentorship, and making sure interns are getting what they want out of the summer.
Have small, well-defined projects ready on day one.
If interns can ship on day one, great. We’ve done it before, we’ll do it again. Well-defined starter projects are the bread’n’butter of team onboarding.
Set aside time in the first two days to explain your team’s higher purpose.
Your team has a big goal for the summer. Bring interns into the fold. They should see the bird’s-eye view of your group’s projects and priorities. If we hired right, they’re smarter than us and will surprise us w/ new ideas.
Set code review expectations.
We care about reviews and the relationships they build. Three simple steps: tell them their first reviews will be brutal, it’s nothing to be concerned about. Review their code. Ask them to review yours.
Require sharing.
Schedule team-wide demos for your interns. Encourage blogging. We err on the side of being overly communicative.

Mentorship is nuanced, and the above is purposefully far from exhaustive. If we’re gonna have the same old shockingly high expectations for mentorship quality that we had when we were a much smaller team (we do!). we’ll need to keep building artifacts like the above for our growing team (we just did!).

The team’s official living document is here. I’m sure it’ll change soon enough. I’m just blogging Version 1 because we like to share. And apologies to Joel for ripping off his title.

Breaking down Amazon's mega dropdown

Wed, 06 Mar 2013 02:08:17 -0800

The hover effects on Amazon’s big ‘ole “Shop by Department” mega dropdown are super fast. Look’it how quick each submenu fills in as your mouse moves down the list:

It’s instant. I got nerd sniped by this. Most dropdown menus have to include a bit of a delay when activating submenus. Here’s an old Khan Academy dropdown as an example:

See the delay? You need that, because otherwise when you try to move your mouse from the main menu to the submenu, the submenu will disappear out from under you like some sort of sick, unwinnable game of whack-a-mole. Enjoy this example from bootstrap’s dropdown menus:

I love bootstrap, don’t get it twisted. Just a good example of submenu frustration.

How did Amazon get away without using a delay?

It’s easy to move the cursor from Amazon’s main dropdown to its submenus. You won’t run into the bootstrap bug. They get away with this by detecting the direction of the cursor’s path.

If the cursor moves into the blue triangle the currently displayed submenu will stay open for just a bit longer.

At every position of the cursor you can picture a triangle between the current mouse position and the upper and lower right corners of the dropdown menu. If the next mouse position is within that triangle, the user is probably moving their cursor into the currently displayed submenu. Amazon uses this for a nice effect. As long as the cursor stays within that blue triangle the current submenu will stay open. It doesn’t matter if the cursor hovers over “Appstore for Android” momentarily — the user is probably heading toward “Learn more about Cloud Drive.”

And if the cursor goes outside of the blue triangle, they instantly switch the submenu, giving it a really responsive feel.

So if you’re as geeky as me and think something this trivial is cool, I made a jQuery plugin that fires events when detecting this sort of directional menu aiming: jQuery-menu-aim. We’re using it in the new Khan Academy “Learn” menu:

I think it feels snappy. I’m not ashamed to copy Amazon. I’m sure this problem was solved years and years ago, forgotten, rediscovered, solved again, forgotten, rediscovered, solved again.

If anyone else on the planet ends up finding a use for jQuery-menu-aim, I’d be grateful to know what you think.

Thanks go to Ben Alpert for helping me understand the linear algebra / cross-product magic Amazon uses to detect movement inside the “blue triangle.” I ended up going w/ a cruder slope-based approach, mostly b/c I’ve lost all intuitive understanding of linear algebra. Sad. Need to watch more KA videos.

Meritocracy and Discrimination in Tech

Sat, 16 Feb 2013 12:23:00 -0800

I’ve been the jerk who denies a candidate their hopes for an internship hundreds of times. That’s not a point of pride – just a consequence of helping build eight (nine?) classes of interns at two small companies that both happen to receive up to 500 intern applications every month.

I’ve obsessed over finding ways to build internship classes full of the best developers in an industry where the phrase “best developers” is so overused it’s become meaningless. In all cases, the male:female ratio has been exactly what you’d expect in tech.

So when Jessica penned her post the other day celebrating the fact that Khan Academy’s inbound Summer ‘13 class somehow has twice the number of females as males, I smiled big. Even though we all knew it wouldn’t last as acceptances come in (it already hasn’t held), our team enjoyed a brief moment basking in the future that we (and all educators around the world) hope to build: a world in which being a female developer ceases to be a novelty.

Here’s the top-voted comment on her post:

Here are some selected comments from sharing it on Facebook:

Here’s John Resig on Twitter’s equally frustrating if less-accusatory response:

Some really disappointing replies to my last tweet about Khan Academy’s improved female:male ratio. imgur.com/tnmgl7B

February 13, 2013

As the person who stands at the end of our hiring process’s pipeline, I find “Mark“‘s idea that we’re sacrificing quality to fill some quota merely very insulting. If I were one of the women who has successfully navigated our brutal interview process, I’d be furious.

It doesn’t help to be an industry obsessed with meritocracy if the first reaction to an altered status quo is that OUR MERITOCRACY MUST BE BROKEN, SOUND THE ALARMS. That’s an a-hole old-timer’s club.

We should celebrate shifts we see in the historically depressing numbers of women in tech, especially when validated by rigorously competitive hiring environments. Why celebrate if we’re aiming for perfect equality? Because we’re not there yet and should encourage change we want to see.

As I was pacing around our company parking lot trying to decide whether or not to curse in this post I kept thinking about the hubbub raised when a 14-year-old posted his iPhone juggling game on Hacker News. He told the community his age. Because of that, he was met with accusations of “emotional manipulation” and “why can’t we leave age/sex/race out of it all.”

It requires a dangerously simple view of the world to decide that we don’t want to encourage a 14-year-old who’s pushing his limits by building mobile apps because we’re scared of risking our oh-so-perfect meritocracy.

It’s not a meritocracy until all 14-year-olds know that they can build mobile apps. It’s not a meritocracy until we don’t have one gender wondering if tech companies are just stomping grounds for another.

So we’ll continue encouraging females, children, and just about anybody to get into tech by celebrating milestones along the way. To those this threatens – oh.

Hacking at a nonprofit is just like its counterpart

Sat, 02 Feb 2013 13:13:00 -0800

Paul Graham claims nonprofit startups are similar to their counterparts. As soon as I read the headline I realized I’ve been answering a different version of this question in every recruiting conversation I’ve had for the past two years:

“What’s it like to be a hacker at a nonprofit?”

I’ll answer candidly based on my stories on each side of the profit divide. I won’t claim objectivity. I hope to prove this is the wrong question to be asking. Great companies share qualities far more important than 501(c)(3)-ness.

For-profit story: joined Fog Creek when there were ~4 employees*. Was there for 5+ years until it hit ~35-40**.
Nonprofit story: joined Khan Academy when Sal’s solo act suddenly became a “real” 5-person company. Now 2+ years later we’re ~40.

Judging by the questions I get asked by candidates, many would think the for-profit vs. nonprofit issue makes for a striking difference in these otherwise comparable stories.

Truth is both journeys have been eerily similar. I’ll spell out the specifics.

Real stick extends far beyond camera frame.

The basically identicals

1. Day to day, you’d never be able to tell a difference. Walk in every day, hack on products, hack on the dev team and its culture, hack on recruiting, battle the hybrid maker-manager schedule, walk out. Realize I forgot my keys. Walk back in.

2. Technical challenges don’t care about your corporate structure. I’ve been stumped at both companies more times than I care to admit.

3. Speed and autonomy is celebrated, red tape and bureaucracy is reviled. The image of a political, slow (or at least less fast) organization is one of the most typical FUDs thrown at nonprofits. I can’t speak for other orgs, but I can guarantee that with the right people at the helm, “nonprofit” != “slow”.

4. Recruiting with a capital R is a cultural cornerstone at both Khan and Fog Creek. Some assume nonprofits wouldn’t approach recruiting with the same fervor and dedication as their counterparts. Not so. I learned everything I know from much wiser coworkers at Fog Creek, and I consider my current team a once-in-a-lifetime experience.

5. Compensation could be at the top of this list of similarities, but I don’t like to emphasize it. If you think working at a nonprofit means you can’t command highly competitive comp, you’re just wrong.

The mentionable differences

1. The lottery ticket doesn’t exist at Khan Academy. Nobody’s getting filthy rich off an exit, and we know it. This is perhaps the most obvious distinction when compared to for-profit startups where nobody’s getting filthy rich and they don’t know it. I keed, I keed!

If you need to have that lottery ticket, a nonprofit may not be for you.

Of course, my Fog Creek experience isn’t very differentiating when it comes to the typical startup lottery ticket. It’s entirely bootstrapped by principled and generous founders who’ve set up a company that immediately shares success instead of pinning hopes on an exit.

2. Mission is baked in at nonprofits. It’s easy to think of a for-profit startup that pivots left and right in an attempt to find traction. For better or worse, it’s harder to imagine a nonprofit completely abandoning its founding mission.

Once again, “Fog Creek-vs-Khan” isn’t very interesting here. Great companies will have meaningful missions whether they’re for-profit or not. Fog Creek is on a mission. Google certainly has one. I wish Stack Exchange would answer all the world’s questions faster. There’s nothing about 501(c)(3) status that dictates whether or not each teammate shows up every day aimed at the same epic purpose.

But there is something unique about bold nonprofit causes like Khan’s or Watsi’s that makes it highly likely you’ll find mission-driven culture lurking in every corner of the company. Sal ends our weekly company meetings by reminding us what Carlos Slim told him about our product’s future possibilities: “Billions of people are waiting.”

Epic mission: sleep 90% of day

The right questions to ask

So I completely agree with Paul Graham. “Nonprofit or for-profit?” is not a very useful question for a hacker trying to decide between two teams.

Far more relevant:

Is the company built by founders and backers who will do whatever is necessary to gather the best team?
Will you be challenged by talented coworkers and given room to show what you can do?
Are you working on a product you genuinely care about?

I’m only one data point. But in my experience, answering yes to all three of these questions will leave you feeling incredibly lucky, regardless of the corporate label.

*Standard “depends how you count” rule applies here.
**My memory sucks. Please correct me if this number is off, some Creeker w/ a better noggin.

Thx @jason for the read-through and tips.

Pending queues and loading requests on App Engine

Fri, 18 Jan 2013 01:44:00 -0800

Sometimes it feels like we’re playing a rousing of guess’n’check when adjusting our App Engine performance settings. The request scheduler is a bit of a black box, so it can be hard to know which knobs we should twist and how far.

But there are two really straightforward signs any App Engine dev should watch out for. Both are a big red flag signaling a configuration not optimized for performance:

requests are spending lots of time in the pending queue
requests are often causing new instances to be created

The dreaded pending queue

You can identify whether or not your requests are wasting time in the pending queue by looking at your request logs and spotting high pending_ms values. Here’s an example:

98.235.99.108 - - [16/Jan/2013:16:03:12 -0800] "POST /api/v1/user/exercises/distributive_property/problems/1/attempt HTTP/1.1" 200 2558 - "Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0" "www.khanacademy.org" ms=877 cpu_ms=649 cpm_usd=0.000000 pending_ms=195 instance=00c61b117c17e24c3f547df294a7e3c9882e9704

If you see lots of that, it’s probably a sign that you don’t have enough instances running to serve your traffic. The symptom is that requests are getting queued up behind other requests, waiting for other requests to finish before even starting to run your code. Hence the pending_ms.

The causes are trickier to spot. If you think you’ve already optimized your number of idle instances and pending latency settings you may want to focus on making your loading requests faster. If App Engine thinks it takes a really long time for a new instance to load your app’s code on the first request, it may be less willing to give you that new instance when a request comes in. Have fun tapping your toe while pending_ms ticks up. You should also be aware of any requests that are abnormally slow and consider moving them to backends or specific task queues. Otherwise App Engine might queue up some normal request like a user loading your homepage (“I swear, it’s usually so fast!”) right behind some long-running bruiser of an API call. And all of a sudden your user is waiting an extra 500 millis just to get out of the pending queue before your homepage can even start to be served.

Loading requests

Loading requests can be an equally bad sign. You want your instances to stay up and stay healthy for as long as possible. They should be serving thousands and thousands of requests before needing to be recycled. Otherwise your users just sit around on their butts waiting for App Engine instances to reload all that beautiful code you’ve spent so much time writing.

These requests are spotted via loading_request=1 in your request logs. They’re going to be slow, so you don’t want to see ‘em much.

If you’re seeing an abnormally high amount of loading_request=1’s given how much traffic you serve, chances are that you’ve also been noticing bigger problems. This could be a sign that your requests are timing out or simply crashing when a new instance starts up. This’ll cause App Engine to kill the instance being loaded, which causes another loading request to be fired for the next request, which…well, you get the point. You might also have memory leaks that are causing instances to get shut down after serving just a few requests. Bottom line is something is causing your instances to thrash off and on too often.

Spot these problems with the mini profiler

As of today, you can spot both of these problems when browsing around your site if you’re using the App Engine Mini Profiler. If you’re wondering why a request is slow, open up the profiler and take a gander:

You should, of course, still keep track of aggregate pending_ms and loading_request appearances via your request logs. The mini profiler just makes it easy for you to take a glance at the current request’s health.

A sampling profiler for App Engine

Sun, 13 Jan 2013 22:08:18 -0800

I’d been getting a bit frustrated trying to figure out where time was being spent during a few of Khan Academy’s longer running requests. App Engine Mini Profiler usually works great, but these specific requests seemed to be spending a lot of time serializing objects into JSON representations. JSON serialization, by its nature, involves tons and tons of nested function calls. Anathema to an instrumented CPU profiler like the Python profiling tool included in Mini Profiler. So I added a simple sampling tool.

I am Fleetwood, the new star of this blog.

The problem with instrumented profilers

The ugly problem w/ instrumented profilers is they overestimate how much time is spent in frequently called functions or those with deep call stacks. Since “instrumenting” your code basically means turning this…

def monkey_print(s):
    print("ooh ooh aah %s ahh" % s)

…into this…

def monkey_print(s):
    profiler.start_timing("monkey_print")
    print("ooh ooh aah %s ahh" % s)
    profiler.stop_timing("monkey_print")

…you can imagine how all those calls to profiler.start_timing and profiler.stop_timing will add a lot of unexpected overhead if your code happens to monkey_print thousands of times.

This subtle problem can make it really, really frustrating to figure out where time’s being spent in your program. Even after years I’m still often tricked into chasing mirages. When you look at the output of a request that includes thousands and thousands of deeply nested function calls, you can never really trust how much time was actually spent in those functions.

How sampling helps

Sampling profilers have always been around to tackle this problem. They work by periodically halting your program, grabbing the stack trace of whatever code happens to be currently running, and then letting your code go on its merry way.

By doing this enough and combining the sampled stack traces, these profilers do a pretty good job answering “Where is time being spent in my program?” by periodically asking the question, “Well, what is your program doing right now?”

I’m a really nervous car rider.

Right around the time when I started to think I’d trade my kingdom for a sampling profiler that works on App Engine, I stumbled across a Stack Overflow user who’s been on a bit of a personal crusade to get people to embrace a very simple form of profiling: smashing Ctrl+C at random points to halt your running program, then examining the resulting call stack.

He persuasively argues that even manually gathering a small number of call stack samples in this way does a great job answering the most important profiling question, “Where is time being spent?”

I figured, well, if that’s true, then to get some interesting App Engine performance info we might not need a fully-featured, built-by-the-pros sampling profiler. We only need something that can briefly inspect the call stack of the currently running request.

Sampling App Engine

Unburdened by worries of perfection, the hacks began. Stack Overflow taught me how to inspect the stack of all running Python threads. Alpert gave me the tip of spawning a separate thread to periodically perform these stack inspections (since the signal library that I’d prefer to use is off-limits in App Engine).

A little duck tape here and some apologetic comments there, and a simple sampling profiler for App Engine was born.

Here it is. Far from perfect, needs more work, but we’re learning from it already.

Give it a shot

The sampling profiler is bundled into App Engine Mini Profiler along w/ the instrumented profiler and RPC statistics it already included. Profile your App Engine app and let me know how it goes. Or try the (intentionally slow) demo here (just click the little profile link in the upper left corner).

A trail of zebras in my wake.

P.S. We’ve been getting in the habit of giving thanks at the end of Khan Academy company meetings. Whenever I talk about the mini profiler I’d like to send thanks to the Stack Exchange team for their inspiration. If you’re on .NET or Ruby, stop wasting time and check out their original. Given the timing of recent events, I’d also like to send our team’s thanks to Aaron Swartz, the brilliant creator of web.py, which inspired the design of App Engine’s webapp, which, in turn, helps power Khan Academy.

"I don't understand"

Sun, 16 Dec 2012 15:19:00 -0800

I’ve noticed that the most mature and accomplished developers I’ve worked with are also those who most frequently say “I don’t understand” when they’re listening to a technical explanation. This has been the case with coworkers both at Fog Creek and at Khan Academy.

In one way, it’s counterintuitive. Shouldn’t the senior devs already know everything? But it makes a lot of sense. Those who are most secure in their own abilities are the most comfortable to admit when they haven’t fully wrapped their minds around something. Newer devs assume that their confusion is their own fault. They don’t want to interrupt others due to their own perceived shortcomings.

New devs should try to really get just how common it is to not fully understand a technical problem. Most tech stacks have crossed the threshold of one person being able to hold the entire codebase in their head, especially at companies that are hiring. And once that threshold’s crossed, you’ll start hearing from people about the new javascript rendering framework or the latest mapreduce pipeline or a bug in the deploy script or a proposal for a new caching pattern, and a little voice in your head is going to start saying, “Wait…I don’t get it.”

“I don’t understand” is the perfect response. You’re not insulting anybody. You’re not showing weakness. You’re building a culture of respect for how smart everybody is, because you know that after a few minutes of explanation you will get it.

Either that or you’ll find a bug. I like to think of “I don’t understand” as a kind of reverse rubber ducking. Except in this version, the duck comes alive and quacks and stomps and “I don’t understand”s all over your keyboard while forcing you to explain various things.

“I…don’t understand.”

It’s most said by the best, decades after they’ve become a master. We newer devs should follow their lead and get rid of any stigma associated with those words.

Dangers of using memcache counters for a/b tests

Sun, 25 Nov 2012 19:31:19 -0800

We ran into a hairy problem with our App Engine A/B testing framework a few months ago. It’s a problem unique to those relying on memcache to increment counters with statistical meaning in a shared resource environment like App Engine.

Since there are more live-action teenage mutant ninja turtle films than people who fit that bill (sorry, one person on the internet, I know you matter), this is more a story about an interesting problem than a service post.

The symptoms

Some of our A/B tests started looking a little…surprising. “500+ users mastered Dividing Decimals in 20 problems or less, and 200+ users mastered it in 30 problems or less,” our dashboard claimed. Hmmm. Impossible, inconsistent results like that started popping up all over. I had that awful feeling you get in your gut when you square yourself to the possibility that the metrics you’ve been using to make decisions are straight-up broken. Trust in our A/B experiment results plummeted.

Not ideal.

The problem

Our A/B framework uses memcache as the fastest way to increment its internal counters every time somebody new participates in an experiment or triggers a conversion. For example, when a new user first comes to our exercise that helps build intuition for one step equations, they may be handed one of two alternatives for each problem they encounter. A counter (alternative_A_participants) is incremented accordingly. Another counter keeping track of conversions (alternative_A_conversions) is incremented similarly when we think they’ve mastered the skill. This is how we keep track of our A/B alternatives and their relative performances.

Now, there are two properties of these counters that aren’t negotiable. They must be fast, because A/B testing shouldn’t degrade performance for users. And they must be atomic, because otherwise we’d lose tons of data when we get hundreds or thousands of new participants per second, say during traffic events.

In App Engine land, memcache is the clear winner when in need of a fast, atomic counter. You could build a sharded datastore counter for atomicity, but you’ll sacrifice speed because the datastore can’t keep up with memcache. You could try something like kicking off a task queue for every increment, but you may quickly breed a single user’s HTTP request into 5 separate requests for your instances to swallow. Other tools exist, of course, but they aren’t available in App Engine.

Turns out memcache.incr is hard to beat.

The astute among you are already concerned. “Memcache shouldn’t be used to store data you can’t afford to lose,” you chide, “you never know when memcache will decide to evict your data”. True. In a situation like this, you have to be constantly running a process that persists data from memcache into something more permanent, like the App Engine datastore. We do. But you still don’t have any guarantee that a malevolent memcache won’t evict unsaved data before it gets persisted. So you also have to be willing to occasionally lose some data to bad luck. We are.

We knew that we’d occasionally lose count of an experiment’s participant or two or three or twenty. This is a relatively rare problem we’re willing to swallow. Assuming there is no statistical bias in memcache’s tendency to evict the counter for alternative_A_conversions vs. the counter for alternative_B_conversions, we felt comfortable comparing an experiment’s alternatives and making meaningful decisions.

That assumption was way off. Think about the problem that a PaaS like App Engine has to solve when offering memcache to all its applications. They can’t just run one big, honkin’ instance of memcache on one bigger, honkin’er machine somewhere. They have to distribute their load across multiple completely independent instances of memcache. Which means when you call memcache.incr(key), a value could be incremented on one of any number of different App Engine memcache instances…and the instance chosen depends on a hash of the key being incremented.

In simpler words, memcache.incr("alternative_A_conversions") stores its counter on a different machine than memcache.incr("alternative_B_conversions"). Since memcache is a service App Engine shares among all its applications, it’s highly likely for one memcache machine to be under considerably different memory pressure than another.

Streak and Pulse are two heavy App Engine users with whom we’ve probably shared memcache space.

Long story short? Some memcache keys (alternative_A_conversions) can survive in memcache for 10 minutes before being evicted, giving our background persistence task plenty of time to run. Others (alternative_B_conversions) may only last 10 seconds under high pressure. This behavior is consistent for each key…creating a statistical bias that favors one A/B alternative over another by repeatedly evicting only one of the key’s counters before we have a chance to persist its data.

The solution

What to do? We could switch away from memcache for our counters, but as mentioned above the alternatives aren’t great. We could give up on our realtime dashboard for A/B results, stop worrying about atomic counters entirely, log events for every participant and conversion, and periodically analyze our A/B alternatives offline. This may be the future for us, but we’ve been pressed for time (pretty sure we’re unique in that respect) and hoped for a quicker fix.

We had one hack available to us thanks to App Engine’s treatment of the namespace parameter available in all memcache calls. If you provide a namespace in your call to memcache.incr, App Engine would use that namespace to determine which memcache instance to send the value to, *not* the key. This meant we could keep all keys for an experiment in a single memcache instance, which would remove the statistically biased eviction patterns. While I’m normally a lover of such simple solutions, we heard this would soon be changing, and even identically-namespace’d keys would be spread among multiple instances in coming weeks. No go.

Luckily, Google’s Fred Sauer is brilliant and had a better idea. He suggested using bit offsets to store multiple counters’ values within a single memcache key. Since memcache.incr’s ints have 64 bits, we could split them up into 4 16-bit counters, each one counting up to a maximum of 2^16-1=65,535 before needing to be persisted over to the datastore.

This ended up working beautifully. Each time we want to increment alternative_A_conversions, we grab the memcache value stored by key all_conversions and increment the counter stored in bit positions 0-15. When we want to increment alternative_B_conversions, we grab the exact same memcache key and increment the counter in bit positions 16-31.

Even the most trivial bit twiddling makes web devs like me feel all hardcore. It’s a little sad.

Using a single memcache key for all of an experiment’s counters means we never have to worry about statistical bias in eviction policies affecting our ability to compare alternatives A and B. If one of an experiment’s counters is evicted at an unfortunate time, they all are.

Our A/B experiment data has looked consistent since this change, and trust in the framework is largely restored. Hats off once again to Fred and the App Engine team.

Khan Academy Internship, Summer '12

Mon, 19 Nov 2012 07:32:00 -0800

Teaser: we’re full steam ahead hiring for the summer ‘13 class of interns. Apply.

We had fifteen interns this summer and around that many full-timers.

It’d be silly for me to try to write a summary of what was accomplished. That task was doable last summer and fall, but this time too much was built. If I made a list of it all, I’d ramble even worse than normal.

Luckily, some interns took up the mantle (minus the rambling). If you’re here to get a feel for the type of work done by a Khan Academy intern, you can’t do better than Dylan Vassallo’s, Ben Alpert’s, Jamie Wong’s, Omar Rizwan’s, and Ankit Ahuja’s firsthand summaries. From launching a brand new way to explore computer science (*cough*alongside John Resig*cough*) to shoring up our backend infrastructure (*cough*alongside Craig Silverstein*cough*), these posts give a taste of how our interns spent their time (not apologizing for namedropping, sorry, it’s relevant).

But just a taste.

So before things gets any foggier in my head, I’ll share the summer themes that stick out most for me.

Theme #1 from the summer: did things? Don’t forget to tell people

I prefer to just tell people to “BRAG!!!” so often they think I’m more repetitive than Inigo Montoya, but I suppose “do things, tell people” does have a less abrasive ring to it.

The intern posts linked above create immense value…for Khan Academy, for wide-eyed future KA candidates, and especially for the careers of their authors (Alpert and Vassallo are already full-timers, though, so mitts off). They’re one of my absolute favorite parts of every internship.

But they don’t tell the full story.

On the issue of gender, they actually warp it. Four of our fifteen interns were female, and they each made serious dents in the world. Heck, Jessica Liu single-handedly created an entire library of computer science content which is now teaching hundreds of thousands of learners. It was the most gender-balanced group of interns I’ve ever worked with, and I’m more than convinced this is a big reason why it was also one of the best.

Just one example of a story easily lost. It’s important enough to let stand on its own.

The top student-created spin-offs of Jessica’s Nyan Cat program

Far more important stories go untold than boring stories are shared. My rambling blog may often be on the wrong side of that battle, but hopefully you and your team won’t be. We use dedicated intern demo days, hipchat rooms for the purpose of sharing screenshots, and “You Killed My Father, Prepare to Die”-like persistence when encouraging interns to share.

Theme #2: always more mentorship

I haven’t met an intern yet who complained, “I’m getting a little tired of all the mentorship around here.” I hope I do one day. No matter how much we focus on mentorship, it requires constant, conscious effort. Making sure all mentors are communicating well. Making sure interns are having a consistent code review experience. Making sure ownership is being handed out. Trying to learn from any and every frustration voiced by a previous summer’s intern. It’s a full-time job for every dev on our team, and they already have full-time jobs.

But it’s worth it. See below. You can bet we’ll be gathering next summer to discuss how to be even better mentors.

Theme #3: shipping is good for what ails ya’

Our interns ship to real users. Whether it’s from deploying quick fixes on day one or sprinting all summer for a splashy launch in August, they will have the unmistakable feeling earned by sending their creations off into the world.

I honestly believe this is one of our biggest competitive advantages when recruiting. One of the most talented interns I’ve ever worked with told me about his previous internship’s code — it’s still rotting in source control somewhere. The first week of his summer, he told me that wouldn’t be happening again. I have immense respect for that.

~~Want.~~Need.

There was another intern, one we lost in a recruiting battle to a big company. He emailed me a week after he started, full of regret, after being shoved away in a corner to work on some internal-only documentation tool. What a waste.

I’ve seen frustrations of all varieties melt away when a dev, full of pride, pushes a new piece of work out to hordes of hungry users. It’s cathartic and a capstone experience in any real-world development effort. To not include it in an internship is a crime.

Our team can proudly say we nailed this theme this summer.

We’re all in again

Most tech leads or managers that I talk to think a 1:1 ratio of full-timers to interns is simply too crazy. Maybe.

One thing we know for sure? Working with interns blows the normal interview process right out of the water. As Desmond so rightly pointed out recently, if you’ve ever had an argument about whether or not to give an intern a full-time offer after three months of working side-by-side with them, you’ve really gotta question the effectiveness of your normal, oh-so-vaunted five-hour interview process.

And it’s well understood now that the best devs are only on the job market two, maybe three times in their lives. One of those is right out of college. For long-term recruiting health, you can’t beat internships. The new full-timers joining our team from the class of summer ‘12 are a testament to that.

Punchline? We’re hiring up again for next summer. We’ll be even better prepared and accomplish even more than last time. If you’ve read this far and want to be a part, apply for an internship and include “I read Kamens’s blog, and the monkey swings at midnight” — you may squeak through resume reviews a tad faster.

A maker-manager's schedule

Sun, 11 Nov 2012 08:05:00 -0800

Paul Graham’s Maker’s Schedule, Manager’s Schedule meant a lot to me. It meant a lot to anybody who’s ever been inexplicably frustrated by a kind-hearted coworker interrupting them, or felt like an ass for turning down a request to grab a 20 minute coffee with an acquiantance.

It meant a lot because it explained the frustration. It justified (as much as can be justified) the ass-like behavior. And it did so by drawing a clean line between people who need completely interruption-free schedules (makers) and those who context switch from one block of work to the next (managers).

Truth is, though, that in the startup world we’ll be forced to straddle that line. For many makers and managers, the separation won’t be so clean. Understanding how to navigate a hybrid maker-manager’s schedule can be critical for personal and product success.

Ask a maker and a manager for something that’ll only take twenty minutes out of each of their days (easy part). Then peer into their souls (slightly harder part) and watch the decision they’re trying to make.

This email was sent to both Ben the manager and Ben the maker,
who are two completely different people. For science.

Ben the manager’s thoughts after receiving this email: this person seems really interesting and nice. Would it be possible to fit this into any of my free time slots? Hmm.

Ben the maker’s thoughts after receiving the same email: this person seems really interesting and nice. Should I sacrifice half of a day’s productivity by meeting them or be a rude ass by saying no? Harumph.

Ok, cool. This explained a lot. For years, I had been rude to people I love without understanding why (I’m still rude but at least I know why). I thought they were just refusing to see how important a lack of interruption was to me…even when their request only took a measly 30 minutes out of an otherwise clear schedule. I was a maker! And not everybody intuitively understood the implications! I emailed the article to my mom and my girlfriend and apologized for snapping in the past.

Fast forward to today. By fate or free will or some sort of sick cosmic joke, I’ve spent the last 5 years trying to become a good manager. Trying to make myself always interruptable, always focused on empowering others, always thinking twice before going off on my lonesome to build something. Like many makers breaking managerial, I’ve been challenged. And at the end of the day, most challenges for a maker-turned-manager boil down to the loss of a maker’s schedule.

Far as I can tell, there are three doors to choose from if you’re a maker becoming a manager:

Door #1: Embrace it. Completely give up the maker’s schedule and the idea that you need to personally make to be valuable. Know, deep down in your heart, that all of your time is now best spent empowering others. I know plenty of people who have been able to do this, and I admire them greatly.

Door #2: Flit between the two. Your schedule is now maker-manager. Convince yourself that it’s more important to help somebody else than to build on your own (cuz it is), but still try to build because that’s what you love. I’ve been peering behind this door for quite a while now. The path is rife with confusion. You simultaneously value yourself by what you personally get done and your willingness to not get things done while empowering others. That’s inconsistent and possibly frustrating, but not necessarily wrong or unnavigable.

Door #3: Be the mythical perfect maker-manager. This door only exists in my dreams. If somebody can tell me more about finding time to reach peak creativity while also being an always-available and aware manager, consider this blog post my way of begging.

Door #1 doesn’t work for me. I have trouble sleeping if I haven’t personally created in some way, even if it’s a tiny bug fix or blog post. I cannot manage well if I become too disconnected from the product. These may be managerial limitations of mine, but so be it. It’s in my bones. I applaud those who open this door, but I’m most at home…most myself…when finding a balance between maker and manager.

And since I seem to have lost the keys for Door #3, I’m gonna share survival tips for those passing through #2.

A Maker-Manager’s Survival Tips

Grumpy like me when the scales start tipping past 90% manager?

1. There is a difference between being willing to be interrupted at any time and letting your schedule be entirely controlled by others.

It’s taken me years to start to understand this. Always saying yes to, “Hey Ben, I need some help” is not the same as always saying yes to, “Can we talk at 3pm on Thursday?” You can’t dismiss the former and still call yourself a great manager. You can reschedule the latter for a different day and still be great.

You can take control of your schedule and preemptively block out time for making. I’ve recently gotten better at this (only took a few years). It’s transformed my schedule from this:

…into this:

I’m still 100% interruptable during maker hours. I’ll still drop everything if anybody else on the team needs something from me to be more productive. But I’ve preemptively stopped scheduled 1:1s, recruiting interviews, product meetings, and anything else that would’ve shocked me out of the coding zone for a precious bit each week.

2. There is a difference between 4 free hours at the start of my day and 4 free hours at the end of my day.

Any maker hours I schedule are deliberately pressed up against the end of the day.

Think about how long it’ll take to make whatever it is you’re aching to make next. Say you estimate 3 hours. What are the odds it’ll take exactly 3 hours? 4 hours? How about 5 or 6? If things go very wrong, could it possibly take a week? There’s a distribution of possible time needed, and while it’d be no shock if that 3 hour task took 6 hours, there’s no way it’ll take 0 hours. The distribution is biased in favor of being late. While rare cases may require a week, no case can take a negative week. This is a big reason why software is late so often and being early is a rarity. But that’s another blog post. As my dad used to say, “Everything takes longer than you think.”

It’s also a big reason why I need a wide open schedule for the rest of the day to truly get in the zone. If things go wrong, I might need that extra hour or two to finish up. If I know, in the back of my head, that my block of time definitely ends in four hours…I’ll struggle to slip into the zone.

Call it a weakness. I need to look out over my day’s schedule and get the same feeling I get when I look at pictures of an infinite pool. Endless possibility. Complete silence from the piece of my brain that pipes up to ask if it’s time for that meeting. Freedom to, if I wanted, slip into focus and find myself surprised that the clock says 11pm.

3. Keep a list of wins waiting to happen so you don’t have to think when a little bit of maker time rolls around.

This is my own personal list of things I can probably accomplish in ~half a day. It’s not our bug list, it’s not upcoming critical features. It’s a secret buffet of ready-made maker meals for any time I start getting the shakes.

4. Organize and take full advantage of fixits and hackathons.

When one of our special dev events rolls around, I burn my managerial hat. Judge me if you want, but during fixit days or hackathons I completely disappear in a cave of code and suffer no shame. I prep beforehand so I can hit the ground running. My last fairly significant contribution to the codebase was during our healthy hackathon, and I still happily milk that memory when I need to remind myself of personal creations.

Tuesday’s the next Khan Academy fixit, and I already can’t wait to make. It can be difficult to find the groove as a maker-manager, but with a team like Khan Academy’s, it sure is sweet when you do.

If you have other tips, I’m dying to hear ‘em.

What traffic from 60 Minutes looks like

Mon, 03 Sep 2012 11:47:00 -0700

Yesterday evening Khan Academy was featured on 60 Minutes. It was a rerun.

Ben Eater recorded our real-time active visitors during the East Coast feed
— here it is sped up 10x.

Last time

In March, when the clip first aired, I had Camtasia open and was recording my view of Google Analytics’ real-time visitor data. I was excited to share a view into the traffic Sanjay Gupta commands when he’s not neurosurgeoning or professoring. So the segment starts, we see 10k simultaneous visitors…12k…20k…30k…45k…50k…I test a quick load of the homepage…crash. Try again, crash. Fast forward three fire-fighting hours later, and I would barely remember that I left Camtasia recording the whole thing. I wound up with more video than anyone would ever want of me tabbing around like crazy, trying to help our site scale.

We wound up dropping at least 30% of the East Coast feed’s homepage requests on the floor that night. Rough. We barely managed to patch things up before the Mountain and Pacific showings.

A second chance

Needless to say, we were excited when 60 Minutes gave us the heads-up earlier this week that they’d be running the piece again. We started twisting all the pre-traffic dials we learned about during our struggles last time (tech details are out of scope for this post, lemme know if you’re interested) and crossed our fingers. Our site has changed a lot since March, so we were obviously at risk of unknown scaling issues. But more importantly, Craig, Chris, and Dylan spent the summer battening down our App Engine hatches with a focus on performance and reliability.

Yesterday’s requests per second

The steepness of this spike isn’t helped by the fact that this aired on the
Sunday before a holiday, which means the rest of the day had very low traffic.

Things went quite well this time. We were shy of perfect — right at the peak, we dropped about 500 homepage requests due to a slight problem that we’ll know how to prepare for next time — but handled the traffic much, much better than last time. All in all, we saw over 1.2M pageviews from the start of the East Coast feed at ~4:30PM EST to the end of the West Coast feed at ~8PM EST. That’s over 1/3 of the pageviews that POTUS got on the day of his reddit IAMA — a cheap, unfair, ridiculous comparison since we’re counting pageviews across our entire site vs. a single reddit post, but, hey, politics aren’t fair. Again, much credit to App Engine and our App Engine team. All I did was yell out “50K!” ceremoniously when the concurrent users counter ticked past fun milestones.

Last March’s Sunday visits:

Our visits peaked much higher during the original airing…

Yesterday’s visits

…but the rerun still delivered quite a bit of traffic for us.

Putting 60 Minutes in perspective

On August 14th we launched our new computer science section of Khan Academy. It was featured by venturebeat, Techcrunch, Wired, Forbes, and more. John’s blog hung at the top of Hacker News for almost an entire day, and twitter was exploding with positive feedback. Despite being a massively successful launch, that amount of online press didn’t come close to the traffic that 60 Minutes has at its fingertips. It’s not a perfect comparison since John’s blog likely soaked up a lot of traffic, but it’s still interesting.

In fact, looking back over a couple years of Khan Academy traffic, no press event has even remotely approached the original 60 Minutes airing’s traffic spike. Yesterday’s rerun and Sal’s TED talk are the next most interesting events.

Lessons learned A/B testing with GAE/Bingo

Sun, 29 Jul 2012 09:20:00 -0700

Almost a year ago we released a version of Patrick McKenzie’s A/Bingo, built for Khan Academy and others in need of an App Engine-optimized split testing tool. It’s been in constant production use at Khan since then. We’ve made mistakes, learned lessons, and spent plenty of time scratching our heads pondering results.

Thanks to those lessons, the current version of GAE/Bingo is a lot more powerful than it used to be. And I’m in a sharing mood.

Lesson learned, tool updated: make it trivial to observe many metrics

We use GAE/Bingo to keep an eye on tons of different metrics whenever we’re testing a change. For KA that means tracking things like exercise problems completed, proficiencies earned in math topics, videos watched, return visits, registrations, and anything else that Jace and his team decide belong in our collection of core metrics. So it’s not rare to spot us yelling “Bingo!” (A/Bingo’s cute lingo for an A/B conversion) at all sorts of interesting points throughout our code:

def attempt_problem():
    # Yelling "bingo!" is A/Bingo's cute lingo for a conversion
    gae_bingo.bingo("problem attempt")

    ...

    if earned_proficiency:
        gae_bingo.bingo("new proficiency")

These bingo()s stick around; we rarely have to change them. That means whenever we run a new experiment, all interesting metrics are tracked independently and automatically with one line of code:

if gae_bingo.ab_test("new homepage layout"):
    # ...try new homepage layout

…so we can ship that line of code, put on our lab coats, open up GAE/Bingo’s dashboard, and start poking around the incoming data. We don’t have to think about how to track all these conversion metrics every single time we add a new test.

Look’a all those metrics just begging to be clicked on.
If you can’t tell, this screenshot is our new homepage layout experiment (which happens to toy w/ an emphasis on search).

Of course, we can also choose to specifically track, say, “proficiencies earned in the Solid Geometry exercise” when tweaking hints for solid geometry — we just add a new bingo() at the right spot.

Lesson learned, tool updated: lean on historical graphs, not stat sig

I’ve covered this before: there are lots of dangers in trusting a single signal of statistic significance sent off from any A/B test, but that doesn’t mean they don’t work. We’ve learned a buttload (shut up, it’s a real measurement) more from our data ever since we started snapshotting A/B test results hourly and graphing them next to our framework’s automatic stat sig analysis. And I’m sure we’ve made fewer mistaken snap judgments.

A single “99% significant” number doesn’t really capture the full story of an A/B test.
This particular test is keeping an eye on the effects of asking users an interactive question
in the middle of watching a video.

Lesson learned, tool updated: leave a trail of your past experiments

We’re currently running 9 different A/B tests, each tracking somewhere between 10 and 35 different metrics. That’s been pretty common for the past few months. A large body of knowledge about our past experiments is accumulating. As our team has grown, it’s become more and more important to figure out how to easily save and share past test results.

The latest version of GAE/Bingo lets you archive your experiment with notes and, because we like a little spice in our software, a choice of emotional reactions to the outcome. Take a look:

We like keeping stories of old experiments around, even if the data was confusing.
This particular test was our attempt at encouraging students to use interactive hints when trying to solve difficult problems.

Our list of archived experiments tells a story about what we’ve been trying to improve.

Lesson still being learned, tool not updated: interpreting results is very hard

This post about puzzling A/B test results at Microsoft came at just the right time for us. Anybody running A/B tests should sit down and read it immediately (anybody not running A/B tests has bigger fish to fry).

Even with all our data, a healthy number of users, and some fancy tools, it can be really hard to understand what is happening when looking at the results of an A/B test. The linked paper puts it best:

…the devil is in the details and the difference between theory and practice is greater in practice than in theory…It’s easy to generate p-values and beautiful 3D graphs of trends over time…real challenge is in understanding when the results are invalid, not at the sixth decimal place, but before the decimal point…Generating numbers is easy; generating numbers you should trust is hard!

I’m ok raising my hand and admitting that we’ve seen some really confusing numbers from our experiments. We’ve seen an experiment involving interactive physics videos increase the number of math skill proficiencies earned…on a completely separate part of our site. We’ve had a homepage experiment drastically impact user registration numbers…but the registration effect started 4 days after the experiment began. We tested a “Send Sal thanks!” button designed to distract users from adding “thank you!!!!” comments under our videos…and users started leaving more comments. I could go on and on.

I’m even ok raising my hand and admitting that these discrepancies caused our team to run one of those (brilliantly named) A/A tests. It has been very humbling and healthy to compare two identical versions of Khan Academy, watch the results come in, and wonder what stories you’d be explaining to yourself if these numbers were tied to a real experiment. If you’ve ever spent time looking at A/B test results, I cannot recommend running an A/A test highly enough. It is a good gut check reminding you what your mind does when interpreting results.

Sure, some of these discrepancies are almost certainly caused by bugs in our code. Some are caused by the fact that GAE/Bingo doesn’t do a good job handling outliers yet, so a single participant can act all crazypants and make our graphs go wonk. But sometimes it just takes us a long time to figure out what the heck’s going on. I’m glad Microsoft published their paper admitting the same.

Grab the tool, start a test, tell us what you learn

Please let me know if you’ve learned any lessons using GAE/Bingo elsewhere. Or if you have tips for improving more. I’d love to share.

How we ran the first Khan Academy Healthy Hackathon

Sun, 08 Jul 2012 20:58:00 -0700

This weekend was our first hackathon. Despite not knowing WTF we were doing as organizers, I’m calling it a huge success. The creations demo’d yesterday blew me away (more to come on that).

When we started trying to puzzle out the few details that need to be puzzled out when running a tiny, private hackathon, we stole plenty from the shared experiences of others. Figured I’d return the favor in hope of somebody stumbling across this post and finding inspiration of their own.

Since this wasn’t open to the public, our planning, advertising, and rules were all bundled up into one email. It’s copied below, but I’ll start by highlighting the unique parts for you tl;dr’ers.

Our hackathon was healthy.
The whole red bull and no sleep thing (and sometimes alcohol, apparently?) doesn’t really align with Khan Academy’s tradition of investing in employees for the long-term. We shared salmon for dinner and ate the best snackies Whole Foods had to offer. We forced everyone to go get sleep at 11:45pm and re-opened the doors at 9:30.

Our definition of “hack” was not limited to code.
Jessica’s not “officially” a dev, but that didn’t stop her from creating a gorgeous new careers page and more. Others from outside the dev team were hacking on copy, improving the names of our math exercises, building useful spreadsheets, and generally creating anything they could think of. Involving everyone in the company was the most unique part of this weekend, and multiple devs told me it made the experience for them. Anybody can fix anything.

We had one “roaming hacker.”
Ben Alpert designated himself the official roaming hacker, ready to teach anyone. He somehow found time between his creations to help almost every other team. After 1.5 internships and a year of working part-time while at CMU, he knows the codebase as well as anybody, and he was always available for interruption (unlike me…I selfishly seized the opportunity to squirrel away and code). When the bell rang Saturday afternoon, multiple hack teams felt indebted to Ben. Maybe next time we’ll have two.

Everyone had to demo.
Anything hacked together had to be demo’d to everyone else. Share your work.

If you’re actually planning a hackathon soon and looking to get a feel for more of the nitty gritty, our kick-off email follows.

To: all@khanacademy.org
Subject: W-w-wild and crazy hacks

♤✌☭☂♘ (The First) Khan Academy Healthy Hackathon ♘☂☭✌♤

We’re having our first team hackathon at the start of next weekend. It’s open and friendly for *everybody* in the company, regardless of whether or not you currently think you’re a hacker, and kicks off Friday afternoon.

WTF is a hackathon?

Have you ever sat quietly at home and thought to yourself, “I’d really like to make some even nicer social sharing features, but that’s never at the top of our todo list.” No? You’re not Desmond?

How about, “I’d love to experiment with a simple fact recovery exercise for teaching things like multiplication tables, but it’s just not critical.” No, not Jace?

Maybe, “I’d love to make sure all our code has unit tests and docstrings and isn’t filled with ugly circular references and generally appears to be written by semi-competent professionals.” K, you’re not Craig…

This first hackathon is a chance to work on absolutely anything you want that’s connected to Khan Academy. If you have an itch, scratch it. It doesn’t matter how wild or crazy it is. Hence the Omar Gooding-themed email subject.

WTF do I make?

We just covered that. Anything related to Khan Academy. Here are some possibilities cheesily accentuated with exclamation points:

Make your first interactive math exercise!

Use Khan Academy’s API to integrate with a cool external service like DuckDuckGo via http://duckduckhack.com!

Make an exercise that teaches people how to use sign language for www.khanacademy.org/labs!

Run an A/B test or three on the knowledge map to figure out what wording gets people to start the most suggested exercises!

Drastically change the world in less than a day by building a quick landing page for our translated videos!

Write a blog post about what you do!

Analyze our data and try to figure out if users from a specific non-US country have unique usage patterns, then blog about it!

Make a little widget that influencers can drop on their blogs to link back to their public Khan profiles!

Learn a new programming language by creating a wrapper for our API in the language of your choosing!

Create a better blogging system for our team! (props Jessica)

…I’m sure you can think of more that don’t need the cheesy exclamation points to sound cool. They go here: https://trello.com/board/khan-academy-healthy-hackathon/4fed76fc065593df077466de

Do I have to know how to code?

NO. Hacking is about solving a problem and building anything. It’s perfectly fine to treat this as a chance to learn (take a crack at an exercise!) or create something other than code. Khan-like videos, exercise specs, blog posts, beautiful whiteboard murals…as long as you’re creating something that you can show off to the rest of the group, you’re hacking and you’re in.

Ok, so WTF do I do now?

Add your ideas to the Trello board. https://trello.com/board/khan-academy-healthy-hackathon/4fed76fc065593df077466de

Commit to an idea if you’re sold.

Start lobbying others to join your team if you want. Humbly suggest using this as a chance to work with people you don’t work with every day.

Prepare to hack with a mixtape or two blaring in the background. Do not start coding yet. David.

WTF are the rules?

You have to create something connected to Khan Academy.

You have to demo or show off what you’ve created at the end of the hackathon.

You have to act like a healthy hacker (sleep, eat good food).

Details please.

Start
5:30PM on Friday the 6th. Our office. We’ll start by letting idea owners pitch to others and try to convince them to help.

Intermission
We will be locking everybody out of the office at 11:45pm on Friday. You will go home and get a healthy night’s sleep. We will take your computers if necessary. David. Doors reopen at 9:30am on Saturday.

End
Coding stops at 5pm on Saturday. Keyboards will be disconnected. Every team will then demo — expect to end around 5:30.

Food
Dinner will be provided Friday night and there’ll be snacks available all day Saturday. Lunch is up to you. Dinner and snacks will have a healthy theme.

Teams
We highly encourage hackers to work in teams, but keep the size limited to
Prizes
The judging rules are secret, but prizes will include a [redacted], a [redacted], a [redacted], and a [redacted] :). Also one or two other things.

Do I have to attend? This is my weekend. FOR THE LOVE OF THE GODS LEAVE ME ALONE I NEED TIME IN MY ZEN RELAXATION STUDIO.
No problem. We believe the weekend is your time, and you should use it without hesitation. This is a completely optional healthy hackathon that just happens to instantly make all attendees ridiculously cool and good-looking. The choice is yours.

There will be a next time. We had a blast. Things I’d change? …add another roaming hacker, rethink how prizes are awarded, and consider adding another day to support even more grandiose hacks.

My Compute Engine Pipe Dream

Mon, 02 Jul 2012 07:42:20 -0700

I’m gonna don my rose-colored glasses and take a look at Google’s newly announced Compute Engine. As the majority of Khan Academy runs on App Engine, I’m very excited about a specific possibility here.

While I’m aware of the fact that Compute Engine’s list of use cases is currently aimed at backend data analysis type stuff, I’m gonna take a guess and say that it’s only a matter of time before they add the necessary tools to start serving user-facing web apps.

From http://cloud.google.com/products/compute-engine.html —
I doubt someone tripped and typed “(Initial)” accidentally.

They’ve already got App Engine pointed squarely at the application problem, so it makes sense to initially roll out Compute Engine without worrying about that use case. However, to really compete with EC2, Google’ll have to take this next step. App Engine is a great product, but it’ll never provide the flexibility of access to a virtual machine — and that’s the exact need that causes many shops building user-facing apps to choose AWS. Google engineers are smarter than me, and they know this. Unless I’m way off base, It’s only a matter of time before Compute Engine steps in.

Here’s where things get interesting. Google’s datastore — the one already in use by App Engine, the one touted as the world’s largest public nosql datastore — is unique. Unless I’m wrong — and god, I’d love to be wrong — there is no datastore solution out there that comes close to its scalability + querying power with such shockingly few management concerns. To put it more precisely: what other datastore, database, nosql, sql, tape drive, or box of chalk can you drop into an application that’ll permanently persist terabytes of data with thousands of writes per second without forcing you to think long and hard about sharding your data across multiple machines? You might answer Amazon’s DynamoDB, which is becoming pretty cool and can handle all the writes, but the queries and indexes enabled by Google’s datastore are much, much more powerful. I’m sure Amazon is going to continue adding features to DynamoDB, but right now Google’s queries win easily.

Our App Engine datastore is currently using 7 terabytes of space. Before writing this blog, the last and only other time I needed to check that number was when I was creating a trivia game out of random Khan Academy facts.

If it’s not obvious by now, my pipe dream is for Google to combine Compute Engine’s flexible virtual machines with App Engine’s ridiculously impressive datastore. It’d need to be tight, automatic integration, in much the same way the current datastore only requires App Engine developers to include a Python library or two. It’d need to have latency comparable to App Engine’s current datastore latency, which presumably means you’d need a guarantee that your Compute Engine virtual machines have fast connections to all the right datastores.

I might be talking about science fiction. For all I know, there is some core issue that makes this type of integration a complete nonstarter. But if it ever exists, Amazon will have to play datastore catchup and many shops will have to take a good hard look at Compute Engine when preparing to build an app that needs flexible virtual servers and a powerful, scalable datastore with insultingly low maintenance requirements. We certainly would.

App Engine Performance Essentials You'll Certainly Need, Part II

Wed, 23 May 2012 02:05:00 -0700

Update: the bad fetch performance described in this post has been fixed by the ever responsive App Engine team (see comments below). However, this read is still useful and I still recommend using .run instead of .fetch to stay on top of current App Engine best practices.

It’s kinda cheating to call this Part II of App Engine Performace Hacks You’ll Probably Never Need, but I recently stumbled across a single tip that anybody in App Engine land will want to know about.

Don’t use `fetch()` for datastore access, it’s slower than it used to be

If you’ve been on App Engine for a while, you’re either surprised by that tip or more informed than I was earlier today. Until somewhat recently, it’s been the case that running:

for monkey in models.Monkey.all().fetch(500):
    monkey.swing_from_vine()

…would’ve been much faster than:

for monkey in models.Monkey.all():
    monkey.swing_from_vine()

…assuming that you have at least a couple hundred Monkeys (who doesn’t?) and are willing to eat the memory required to hold all Monkeys at once (who wouldn’t be?). That’s because all() returns a query iterator which chunks up its roundtrips to the datastore into a bunch of small trips, while fetch() (it is hoped) only goes back’n’forth once.

Most people who once wrote fetch([some big number]) were
trying to avoid RPC performance waterfalls like this. Now it can cause them.

So if you’re anything like us, your code is probably littered with fetch() calls all over the place wherever you felt comfortable grabbing a large number of entities and were willing to make a deal w/ the App Engine memory gods to get it done as quickly as possible.

This no longer has any performance benefit unless you use the new batch_size parameter.

fetch() and run() now accept an optional batch_size parameter, but fetch()’s is undocumented. You want something like: fetch(1000, batch_size=1000) or any reasonable batch_size for your use case. Without this parameter, fetch() simply returns list(run(...)) using run()’s default parameters, which happen to set a small batch size.

However, since most people were using fetch() to get around the batch size issue and don’t actually need all entities in memory at once, I recommend following the recently updated docs and always using run() with a reasonable limit and batch_size:

# A grossly simplified couple of lines from our student reporting code:
for problem in models.Problem.all().run(limit=10000, batch_size=1000):
    problem.add_to_student_report()

…which protects you from unnecessary round trips and still gives you control over the amount of memory you’re willing to allocate at once. It feels like fetch() is going the way of the dodo in App Engine land. You can see similar hints dropped into the App Engine codebase right when their team made the ‘ole switcheroo:

Critical docstring change right when the implementation switched.

If you’re working in App Engine land, you’re going to find countless examples all over the web of people advocating or just using fetch([some big number])-ish code that directly contradicts the above advice. I’m gonna go correct this Stack Overflow question as soon as I hit Publish. As of this change, all those old pieces of code suffer from the worst of both worlds: high memory requirements and small batch sizes causing lots of roundtrips.

I’ve verified the perf impact on our production server, and just as expected, keeping your roundtrips low can really pay off. While it’s possible we missed some big announcement somewhere, the fact that we just discovered this means there might be somebody else out there who can benefit.

If you have any other requests for App Engine performance-related topics to cover, I’d love to hear them.

How intern mentorship works at Khan Academy

Thu, 17 May 2012 19:00:00 -0700

Three of our twelve(‽) summer interns have arrived. It’s high time I share how we mentor interns before I’m spending all my time swatting sharks and helicopters out of the air. These tips also apply to new full-time hires, of course, but for now I’ve got interns on the brain.

1. Make the mentorship relationship explicit

At some point in a new employee’s first day they will hear some variation of the words, “I’m your official mentor. As we work through your first few projects together, you can interrupt me any time for any technical question, non-technical question, question about the rules of Bang!, or just because you want to order a specific keyboard. Don’t think twice.”

We don’t throw our interns into a room with 5 people and say “You’re surrounded by mentors! LEARN.” If you’re hiring right, they already know how to do that better than you. What they need is awareness of a specific mentor that will unflinchingly act as their outlet for any need, without frustration.

Good hires will find the right balance between diving into their new codebase to find their own answers and asking you for help. Some may need a little bit of guidance. We’d rather err on the side of interns asking too many questions (easily correctable) than them being scared or unsure of who to interrupt (unfortunate culture problem).

I wish this shark was remote control.

2. Code review everything, and have interns do code reviews

Gonna keep this short because I’ve been over this before. Code reviews form a huge part of our mentor/mentee relationship. Warning though! I’ve been bitten in the past when a class of interns wasn’t code reviewed consistently. If a few interns are getting great code reviews on every checkin and others…not so much…, then they’re going to notice and draw weird conclusions. I’ve been part of teams that have suffered from this in the past, and I publicly apologize. I’m less worried about this now that we have a simple review all the code policy at Khan.

3. Prep your product pre-arrival

The most successful intern projects I’ve been part of began with months (no exaggeration) of preparation. At least for Fog Creek and Khan, intern summers are the most suddenly dramatic infusion of horsepower we ever have to absorb. We don’t want to waste a drop not knowing where we’re going. You have 9 months every year to stumble around not knowing what you’re doing…just get it together for the summer.

I’ve been scientifically tracking every intern class and found this to be the most significant variable.

What do you prep? I’ve seen a couple things work well:

If you are planning on using the summer to launch a brand new product or large feature (I recommend this, it’s a blast), have everything solidified from high-level goals to low-fidelity mockups to code scaffolding.
If you have a less defined vision but generally want to improve a particular area, set aside a couple weeks of bite-sized projects long before anyone new sets foot inside.

Don’t worry about overdefining things just yet. This is about making sure people start with a tangible goal to aim for while understanding how that goal fits into your team’s deeper principles. Smart new hires will have absolutely no trouble stepping outside the bounds of your preparations, taking ownership, and persuading you away from all your stupid assumptions. That’s exactly what your culture should be encouraging.

4. Weekly 1-on-1’s

Our mentors have weekly 1-on-1s with their interns to make sure they’re learning what they hoped to learn and are working on the types of projects they’re most passionate about. These days the top interns have tons of opportunity. It’s a battle for the best, so we make sure Khan interns get the most out of their time. If you think it should be the other way around, you’re probably already missing out on the best and it’s highly likely you never started reading this blog in the first place.

1-on-1’s can be just a few minutes and may fall back to every other week when appropriate. Just enough to make sure our interns have a chance to self-direct.

5. Friday tech talks

“Keep learning” is on the short list of Khan Academy company values. Jamie Wong started his internship this summer and dropped this tech talk about meteorjs in the first week. We’ve had tech talks about fonts (typefaces? styles? ugh I can never remember the politically correct word, I’m so sorry Marcos), game design, a/b testing, performance, and how Khan Academy is being used in towns without internet connectivity. We’re inviting cool people like Steve Yegge to teach us about API-centric development. We want to learn more.

In fact, if you like Khan Academy and want to give us a treat, we’d love for you to come teach our team something. Email me - kamens@gmail.com — I promise a low-key atmosphere and maybe board games.

"Step one? FIX. Step two? IT."

Wed, 18 Apr 2012 01:51:58 -0700

“Step three? FIXIT. Now repeat till it is FIXED.”

Wise words from Kenan, patron saint of the first Khan Academy fixit.

Fixit day is just one more in the long list of solid dev lessons I’ve been learning from the Googlers around here. Since I couldn’t find a reputable source explaining the fixit culture (who reads the NY Times?), I figured it’s my duty to share.

Who’s gonna fix it? They are. Because they broke it.

What’s fixit day?

“The goal of a fixit is to address niggling concerns that bother you time and again, but never enough to actually fix them.”

That’s straight from somebody who has more experience with fixits than either of us. I prefer to view fixit day as that moment when you finally wake up in the middle of the night with enough consciousness to properly rearrange the blanket (or swipe away the breadcrumb, if you’re gross like me) that’s been just barely interrupting your sleep for the past five hours, but never enough to break you out of your dream. Man that’s a good feeling. I hate breadcrumbs in bed…but I also really like cinnamon toast.

Anyway, our first crack at the whole thing tackled two levels of fixits:

Two teams split up to handle upgrading us to the latest version of Backbone and refactoring our code to reduce our circular import problem, respectively. These were both nontrivial tasks that would’ve been difficult for an individual to get done in the face of a rapidly changing codebase, so they were perfect fixit fodder.
The rest of us swept away just about any breadcrumb that needed sweeping. Error logs were cleaned up. 404 pages became prettier. Old, unused code was removed. Crashes were fixed. Little CSS bugs were stomped on.

The “fixedit!” Trello column got quite full, fast. And while it’s easy to take potshots at fixit day by claiming that everybody should be fixing this stuff all the time, I was even happier with the team culture inspired by fixit day than with any individual change. Every single person on our team was having fun working on the same thing, and no fix was too ~~small~~ ~~annoying~~ UggggghhhhWhatIsThisBrowserDoing to be undeserving of absolutely anybody’s attention*. Our first fixit day felt very healthy. It was a little chaotic at the end due to our refactoring ambitions, but they were good ambitions that paid off. The next fixit will be even better. I look forward to it.

*Making shit work is everyone’s job, unless you’re the shit umbrella for ever-…wait…nevermind. Too much cursing. This one’s not gonna get past the censors on the official KA blog.

Teacher effectiveness ratings, programmers, and Khan Academy's data

Fri, 02 Mar 2012 08:49:52 -0800

Today I was doing whatever it is I do when I ran across this link from Joel:

NYC teacher “effectiveness” ratings are bogus, and the data prove it garyrubinstein.teachforus.org/2012/02/28/ana…
— Joel Spolsky (@spolsky) March 1, 2012

…and my brain started pattern matching. Replace “NYC teacher” with “programmer” in this tweet, and you’d be in classic Joel on Software land. After all, one of Joel’s self-stated missions is to make programmers’ lives better (you can see his efforts played out in both the principles of Fog Creek and Stack Overflow), and he’s spent plenty of time trying to convince us of the stupidity of using automated metrics to assess programmers.

Before we go any further, here’s the gist of the post about NY’s teacher effectiveness ratings that were recently released to the public: they have major flaws. Read the post — but this chart says a lot:

Every point is a teacher that taught the same subject to two different grades in the same year. Think of a middle school teacher handling both 6th and 7th grade math. The x-axis is their effectiveness rating for one of the grades, the y-axis is the other grade.

You don’t have to stare at the graph long to see a surprising lack of correlation. If you’re effective at teaching 7th grade math, shouldn’t you be effective at 6th? Wait. Before you throw effectiveness ratings out the window, read the comment further down the page that points out the increasing usefulness of the published data as you look across multiple years of teachers’ past. Ok, that makes sense. But still, judging teachers on test score summaries alone is madness. Perhaps they’re useful feedback when given to teachers appropriately, with caveats, and all? Maybe the ratings need some tweaking?

None of this matters if the data is published to the public and uses a single, automated metric to reward or punish teachers.

There’s a good reason Bill Gates tore apart the decision to publish this data. He knows that a single metric is bound to be not only flawed, but, if used as an incentive system, also destructive to both teachers and any attempts to improve the metric. His nuanced argument for a system that combines data and highly trained teachers evaluating their peers sounds pretty similar to the belief that highly technical programmers should be the only ones managing other programmers:

But student test scores alone aren’t a sensitive enough measure to gauge effective teaching, nor are they diagnostic enough to identify areas of improvement. Teaching is multifaceted, complex work. A reliable evaluation system must incorporate other measures of effectiveness, like students’ feedback about their teachers and classroom observations by highly trained peer evaluators and principals.

— For Teachers, Shame is Not the Solution

Again, replace “teachers” with “software developers” up there and you’ll see Bill Gates is making the same crusade he made for developers — protection from the type of overly simplified management incentives that destroy your ability to focus on the tasks at hand when working in a complex, creative profession.

I was lucky enough to step into professional programming at a time when an exploding number of forward-thinking companies were starting to treat and recruit programmers effectively. I was never promoted or demoted based on the number of bugs I created or lines of code I wrote. But it’s clear that wasn’t always the way things worked, and when I was in college I remember reading Joel and Paul Graham, who both stood out as Defenders of The Programmer against destructive management.

This is why we’d never, ever use Khan Academy data to single-handedly “rank” teachers or anything else so ridiculous. Khan Academy data (and there’s a lot of it, we just passed 400 million practice problems done) is to be put in the hands of teachers, for teachers, as a powerful tool that lets them dive deep into their students’ individual levels of mastery. We aim to empower teachers with the best tools available and believe that the only people assessing them should be highly trained teachers who understand the nuances of their craft and work with them to improve. Sound familiar to you, devs?

I’ve never been a teacher, but I do know that I’ll never even consider working for a company that assesses my performance based on a single automated metric. I think I have Microsoft and Google and Joel and Paul Graham and co. to thank for the software world’s culture of respect for both data and individuals. And now it’s really cool to see Bill Gates and Joel taking a similar stand in defense of teachers.

Ten bucks says none of the teachers in The Academy for Software Engineering suffer from a single metric incentive system.

Required code reviews

Tue, 28 Feb 2012 09:07:35 -0800

This is the story of the growing Khan Academy team converting me into a passionate fan of requiring a code review for every single changeset.

Those who have worked with me know that it’s a surprising position for me to take. On the spectrum of “Follows good development practices even if it slows down the product” to “Just ship the thing, code doesn’t matter, only users matter,” I tend to fall…right about…[furiously scribbling]…here:

Even though I’ve long been a fan of code reviews at both Fog Creek and Khan, I never would’ve suggested requiring them for every single changeset, no matter how small. At first glance it appears all P̝̂R̫͙̼̽ͪ̽̋Ö̝̹̿ͬ́̐̆̈CÈ̝̱S̮̜͙̩̠S̹͍̳̖͍̆̐Y̩̟̥̟̘̺̠ͭͫ̔, this is web development, not rocket science, and Hey what if there’s an emergency and Wait are you serious, you want me to review my single-line change to a trivial #comment???

Luckily for everyone, we’ve been hiring smarter and smarter people at Khan who can save me from myself. (What’s that theory about smart people and some lake? Lake Okeechobee, I think. With the crocodiles.) We told our team that we’re requiring code reviews for all pushes a couple weeks ago. Spoiler alert: it’s not that processy, and even a Just Ship It clown like me is already seeing immense value from the experiment.

Croc! No, sorry, wait…log!…Or, no! Wait! Sorry…Croc!

What’s been so great about requiring reviews?

For a team that values code reviews so highly, we were missing a lot of them. Turns out there are quite a few excuses you’ll tell yourself to explain why this changeset doesn’t need a review. By explicitly setting the expectation, everything’s much simpler. Just get it reviewed.
Frequent, digestible reviews. By committing frequently and reviewing everything, we don’t let features build up to huge, threatening diffs that make your nose bleed like the emo kids from Chronicle. Any change that makes reviews more enjoyable will amplify all of the other well-understood benefits of code review that we’ve listed in our public Khan onboarding docs so I don’t have to make this sentence any longer by talking about them. Small reviews are just more fun.
Those quick one-line reviews aren’t actually a problem. They take 30 seconds to review, see #1. If you have a legitimate emergency, push it and get it reviewed later, ain’t no thang. If I believed for a second that this decision would hurt our ability to Get Things Done, I’d be fighting it tooth and nail.
In two weeks of doing this, I feel I’ve improved as a developer more than any other period in recent memory. All props to my reviewers, of course, and it probably doesn’t hurt that I’m actually writing code for the first time in a while. But…still.

All changesets reviewed.

It’s not for every group. I’m not convinced it would’ve been right when Jason and I were hacking together alone. But requiring code reviews has already made for a better product and a healthier team, two things that I personally care about far more than a healthy codebase. That’s just a nice side-effect.

P.S. When working on Kiln, I was adamantly opposed to building code review requirements directly into the source control product. I stand by that belief, and I’m almost certain the Kiln team still agrees. Reviews should be required by your team’s dynamics and strongly encouraged by your tooling, not the other way around.