I just stumbled across Sal’s latest video on glucose, insulin, and diabetes, thought it was cool, and checked the Khan Academy homepage to make sure it was listed. It wasn’t. So I hit our magic khanacademy.org/sync-yoself-with-the-youtubez URL and in a couple minutes all was good. This wasn’t always always possible.

Those who keep up with his video production know that Sal moves fast. He loves learning. When I last visited the team in Mountain View, half the time somebody knocked on our door it’d be an Amazon delivery guy holding tons of new textbooks. Sal would break open the latest cardboard box, peer inside with a huge smile, and go, “Ooooooooohhhhh quantum mechanics! This one’s gonna take a while!” You kinda get the feeling that maybe he has trouble going to bed when he hasn’t had fun recording videos and uploading to YouTube that day.

This meant our old process of syncing the homepage with YouTube was falling behind pretty fast. First, you had to sync your local dev datastore with the production datastore, then hit some custom URLs on your dev box, wait ~30 minutes, test everything, re-sync in the opposite direction, and commit to subversion to cap it all off. All in all, the process took about an hour and a half with manual interruptions that made it difficult to multitask, especially for me.

Thanks to the 30 minute wait and App Engine’s 10 minute limit on any taskqueue item, automating this process crossed the threshold from “take an hour, write a script” to the dreaded “chunk it up, do the thing in stages, add new videos but keep them hidden on the main site until they’ve been completely synced, keep some logging around, blah blah blah.” The old me would’ve avoided this work and kept trudging through the grind because there’s always more “important” stuff to be done and who has the time.

But the old me is stupid, so the new me took the time to automate. Now a Google App Engine cron job hits a URL that fires off a chain of taskqueue items:

1. Start logging a new YouTube sync attempt
2. Grab all video and playlist data from YouTube, keeping everything hidden until done
3. Run through all the new data and make readable URLs for every entity
4. Set all the videos to visible
5. Index all video and playlist title and description content for our site’s search
6. Regenerate memcached HTML chunks of playlists for the homepage

…and each step logs its attempt so we can debug. Videos are quickly and consistently synced with YouTube without wasting developers’ time.

Like a lot of things we do around the Khan Academy, this doesn’t qualify as dev rocket science, but it’s a small step forward that makes learners’ lives better.