How We Do Product Development at Quizlet: An Inside Look at the Making of Speller

A few weeks ago we launched our newest study mode, Speller. If you haven't used it yet, go try it. This post is more in-depth and technical than usual, but it gives a behind the scenes look at how we created Speller, our engineering challenges and processes, and how we obsessed over the user experience and the educational experience.

Summary - A Few Of The Lessons Learned (tl;dr)

• When a technical problem is too hard, consider completely different approaches
• Just when you think it's awesome, it could still be way better (keep iterating!)
• Make complex user tasks (e.g., typing accents) easier by showing just what they need
• Visualization is crucial for understanding and tracking student progress
• Don't make it too easy; you'll lose the educational value
• Do usability tests until recurring issues go away and it's a joy to watch
• Loud music helps all night coding sessions!

The Beginning

It started with a simple idea: make Quizlet speak your vocabulary to help you spell. What we ended up with is a product that does that pretty well, but also is an incredibly useful tool for familiarizing yourself with vocabulary for studying a language or anything from your flashcards. Speller is so effective because it combines many "senses" at once. Let's say you're using Quizlet to study Spanish, like we are doing as a company (more on that in another post). You've got a list of Spanish Animals and you use Speller on it. You're going to, all at once:

• Hear your Spanish terms spoken to you with correct pronunciation and a Spanish accent
• See a text definition in English
• See a large image representing the term (it's good to see a tree and immediately think "árbol", rather than having to translate in your head from the English word first)
• Type the Spanish term with your attempt at its spelling
• When you get something wrong, you'll simultaneously 1) see a visualization of what letters you got wrong and how it should have been spelled, and 2) hear the missed word spelled out letter-by-letter for you in the Spanish alphabet, followed by hearing the full word pronounced again.
• You'll repeat the above for each term until you get it correct.

Focus on the Text-to-Speech Audio

Early last year we discovered that text-to-speech (TTS) technology had recently gotten really, really good - not just in English - but in many different languages. After doing a bunch of research on text-to-speech engines and comparing the quality of different voices and engines in several different languages, we launched TTS in 5 languages and it was a huge success with our users. We started working hard to expand to a several of other languages, especially as French, German, and Italian were frequently requested. However, we knew that students around the world were going to be relying on this audio to help learn pronunciation, so we had a very high bar for quality. After getting native speakers from many different languages to test out the audio quality, we ended up launching 13 additional languages with TTS, giving us a total of 18.

There were some technical challenges here:

• integrating TTS engines and voices from different companies and figuring out the technical idiosyncrasies between them
• learning SSML
• figuring out a way to scale to support thousands of users online at once (both technically, and from a cost/licensing perspective)
• finely tuning encoding settings
• getting a "spell out letter-by-letter" mode working in each language.

In the browser, we're using Scott Schiller's SoundManager2 JavaScript library as an interface for having detailed control over audio playback in every browser using either HTML5 or Flash. We learned that HTML5 audio, even in the latest browsers, is currently far from perfect, and we've been able to contribute back to SM2 both by submitting pull requests and finding/diagnosing bugs and edge cases (JSFiddle is great for this kind of thing).

A huge goal for the audio was that it needed to be FAST. That meant:

• producing audio clips from new text snippets as fast as possible
• getting the audio file sizes down as small as possible without losing much quality
• utilizing various layers of caching
• preloading audio clips so that the user doesn't has to wait to hear the next audio clip. (We setup our TTS server to add a fake 10 seconds of latency during testing, and made sure our front-end experience still felt snappy.)

Making accents easy

It's common for people trying out Speller on a foreign language to complain that it's too strict because it marks a spelling as incorrect if you simply forget an accent (e.g. typing an "a" instead of an "á"). We've decided not to be lenient on accents so that we help you learn the actual correct spellings. It's extremely common for foreign language teachers to take points off (or mark as completely wrong) answers in tests and quizzes that are missing an accent.

But we know that for the average student learning foreign languages, often just figuring out how to type "special" characters (á, ç, ü, ¿, etc.) is no easy feat. We've worked hard to make it easier. We wrote some code to look at every character in the flashcard set(s) that are being studied and determine which you might need help with. Then we show you just the buttons that you need - and no more, to keep the UI as simple as possible. (We actually wrote this feature first for Speller, but launched it back in January in the other study modes first).

As helpful as those accent buttons are, picking up your mouse is still too slow when you're going through a lot of words. So for people who were click them with the mouse, we show a little "tip" to try to teach you a keyboard-only faster way to type them.

When a problem is too hard, you might be doing it wrong

Our original prototype of Speller worked quite differently than what we ended up with. In the original, when you incorrectly spelled a term, we cleared your incorrect answer and then tried to get you to enter the correct answer by speaking one letter at a time (but without visual hints). For example, if you paused in confusion, it would say the next letter. If you typed a wrong letter, the keystroke would be rejected.

This seemed like a decent idea for a while (it simulated someone helping you spell a word) but had a number of problems, especially with the implementation. Even after a lot of time writing complicated code using all the different types of JavaScript key events, it was still pretty buggy. We could have continued debugging it, but we took a step back and realized we weren't completely in love with the idea anyways. It was kind of annoying and you ended up just pressing keys until you got it right.

Lesson Learned: Iterate & experiment a lot, but don't spend too much time on an idea that isn't "the one"

After some talking and brainstorming, we realized we wanted to visualize exactly what was wrong with our spelling. That meant we needed to show a color-coded "diff" of the letter-by-letter differences between what we typed and the correct spelling. It needed to happen client-side to be fast, and after finding (and tweaking) Google's JavaScript Diff open-source project, we had something going.

This was a big improvement in making it super obvious when you just used an "e" instead of an "a". But it didn't take long for us to realize it could still be better, and we had the idea to use an animation to "morph" your incorrect spelling into the correct one. We coded this up and loved the concept, and then we obsessed over the details of each of the different parts of the diff animation: should each part slide in from the left or right, fade in, or pop in? How fast should it slide? How much time between each part?

The last breakthrough was to not show the animation off to the right, but to literally make it look like the text you typed in the textbox is what morphs into the correct answer. With some CSS z-index and positioning trickery, the effect worked perfectly! Finally, we made sure that as soon as both the audio "spell out" and the animation diff were complete, it would move you on to try again, but you can hit any key to skip it if you're in a hurry. Here's what we came up with (excuse the video's poor audio).

When have you learned it?

By far, the most difficult part of Speller, conceptually, was the combination of figuring out a) exactly what to test you on, b) how often to test you, and c) how to present your progress on a per-term, per-round, and overall basis in a simple and concise way. We thought long and hard about how to make Speller the most educationally productive that it could be. We started with a few assertions / goals:

1. You should spend the most time focusing on terms you're having the most trouble with.
2. We want you to have get something right more than once before we trust that you really know it. This allows Quizlet to distinguish between you guessing the right answer once versus really knowing something.
3. By the time you've finished, we should be pretty confident that you have really learned all the terms
4. The overall process learning process may take a while, but we want to keep users motivated to keep going.
5. You should be able to visualize your progress in as simple a way as possible

Checkpoint Screen

To accomplish these, we knew we want to break up long periods of studying with frequent short breaks with a "checkpoint" - giving you a chance to breathe and review what you just got right and wrong.

Attempt: "Must get 3 stars per word" **Verdict**: made sense; but visually cluttered end-of-round screen; was annoying to have to do easy terms 3 times; no visual progress during each term.

After some more brainstorming, we came up with a new conceptual model: You have "learned" a term once you've gotten it right twice in a row. Immediately after you get a term wrong, we'll show you the correct spelling and then force you to try again to spell it correctly before you can move on. In this "practice mode" getting it correct doesn't "count" toward getting a "correct streak of 2", but it does have the benefit of really making you take the time to learn how to spell what you missed. This seems to do a pretty good handling the first of the few goals above, but we still needed a good way to visualize the concept to the user as well as indicate their progress.

Attempt: "Show 'before' and 'after' dots representing your streak per term" **Verdict**: It expressed the new conceptual mode, but was way too confusing. Also, people would quickly get to the checkpoint screen but then not click "Keep Going" unless they were highly motivated. An automatic timer would help that.

Attempt: "Slide between different columns" **Verdict**: The sliding animation was nice and conceptually simpler than 2 sets of blue dots, but still confusing, and too much wasted screen space when you started getting a lot of terms.

Attempt: "Overall progress of blue dots, then show latest round's words and their 'dot' status below" **Verdict*: Getting much better! We now have a way to show the user's overall *progress in learning all of the terms, as well as a review of the 7 terms they just studied in that round. But, we did some user testing and people were still confused by the dots.

Attempt: "Add labels!" **Verdict**: Yes! The confusion in users tests with this one went away immediately. People suddenly "got" the conceptual 2 dot model and had a quick way to visualize how their overall progress was, and what it took to "fully learn" the terms according to Speller. This is the final/launched version:

Indicating Progress Per-Term

We needed a good way to indicate per-term how you've been doing. The model of "you must get a term correct twice in a row, and getting a term correct immediately after you get it wrong doesn't count" is not particularly easy to explain, even though we believed it was best. We experimented with a dozen versions of animations and ways to indicate this in a simple way. We tried "animated brains filling up" and crazy Pacman-like ideas:

Ultimately we came up with two blue dots dropping in place or falling out of place as you get something right or wrong. It's subtle enough so that you don't have to think about it at first, but makes sense when users study with Speller long enough.

Indicating Overall Progress

We had decided on using 2 progress bars: one that moved quickly to show progress within a round (bottom), and one that moved slowly to show overall progress on how well you've learned your entire list of words (top). We started with the one of the left and found that while some people could figure out what they were for, they weren't clear enough. Tried experimenting with other ideas and finally decided simply adding labels and reducing the number of colors on the page was the best solution - and it resulted in everyone "getting it" right away.

End of Game Screen

By the end of the game, because you've had to get everything correct twice in a row, so we have a pretty high confidence that you actually know each of the terms. We wanted to give the user some helpful data so they can see their progress and a recap of the terms that they had the most trouble with. In my example below, I only got 66.4% of the prompts correct, however I still "fully learned" all my terms - it just took some time. We did some iteration on this screen as well... but once we had all of the above points decided, the final screen mostly just fell into place.

Usability tests… VERY HELPFUL

One thing we did a much better job on in creating Speller, compared to previous projects, was that we were constantly doing informal usability tests with other people. Generally we'd wait until it was at a point where we were reasonably happy, then we'd find someone to try it out. We tried hard not to cheat, meaning that we asked people to "think out loud" and verbalize questions that they had, but that we wouldn't answer them or interfere if they were confused by something. That way we got extremely useful feedback all through the process. Everyone on our team spent a lot of time helping us and trying it out, but we also paid random students in Starbucks and elsewhere around our office who were willing to spend half and hour with us to try out the latest prototype and give us their honest feedback.

What's interesting about doing testing with real users is that everyone has slightly different pain points / things they are confused by. Many of those are worth fixing, some are okay to write off. But the biggest problems are going to be recurring themes of confusion from different users, and that's where you should keep working the hardest. Usability testing can be painful - sometimes there will literally be a button right in front of their face but they won't see it. After even just a short session we'd come back with a full page a notes we scribbled about little things we noticed that could have been better about their experience.

If I had to sum up our process for creating Speller, it would be: we basically kept iterating until user testing was a joy to watch. We knew it was finally good when we'd find a random person, put them in front of Speller with a set of terms they didn't know, and have them go through the whole thing with very visible progression of learning, and no real pain points or confusion along the way.

Engineering processes & details

The overall engineering processes at Quizlet could use it's own post, but a few specifics on Speller:

• Andrew and I (Phil) built Speller start to finish over several weeks. It was an entirely iterative process happening in code - there was no "spec" stage and the product was not fully designed ahead of time.
• On some features we agreed the whole time. On others we started with pretty different ideas about how something should work, but kept iterating and taking the best of each idea until we agreed on a solution that was usually better than either of us started with.
• The rest of our team was incredibly helpful: they tried out various prototypes and gave tons of great feedback and ideas. Often this feedback was that something was unclear, and it resulted in us hitting the drawing board on various concepts to come up with something better.
• Once we felt the core flow of Speller was conceptually good, we started using GitHub Issues with a "Launch Speller" milestone heavily to track a long list of features, bug fixes, and areas for improvements that pretty much summed up the remaining work to do. Then it was just a matter of crossing them off (automatically with git commit messages) and although for a while, we seemed to add 2 new issues for every 1 we crossed off, we were able to complete all the essentials to launch. Having discrete Issues allowed us to keep track of discussions for specific features and link them to any relevant code.

• Started using Jasmine to do some JavaScript unit testing
• There are almost no images in Speller. We're using CSS3 and JavaScript for most of the nice visual effects. In order to create a good fluid layout, we used the CSS3 box-sizing property to abandon the traditional box model and also got images to auto-resize based on the available width
• Painstaking testing in IE 7, 8, 9 (we have dropped support for IE6) with an attitude of progressive enhancement. Things don't have to be as nice in IE8 as they are in Chrome 14, but it should still be pretty good and fully functional. We did several passes of IE testing and fixing (didn't leave it entirely to the end), but also did big stretches of development without worrying about IE.
• Tried to make Speller fail gracefully: code to detect when some internal state wasn't right (persisting/resuming JS game state when the underlying flashcard data could change is tricky) and showed a nice popup helping the user with the problem.
• We use Google Analytics Event Tracking to observe how far along people get within a game
• Speller consists of approximately 4,000 lines of code - mostly JavaScript.

Launched, but never a final product

The final week of working on Speller was a crazy one! After several nights staying up all night long (blasting Pandora to keep us pumped up), we finally launched at 7am one morning. Later that same day we had the privilege to visit a school where students use Quizlet and got to watch a classroom full of students using Quizlet and observed them trying out Speller for the first time. Though we launched Speller and think it's great, we want to always be looking for opportunities to improve and iterate. A few examples:

• Speller is a lot like Learn mode at a fundamental level. We need to pull some of the best parts about Speller into Learn Mode, and even think hard about how we could merge the two study modes into one über useful way to study. (Note: we started by added a "Don't Speak" mode in Speller for people who love the experience of Speller, but want the difficulty of Learn Mode.)
• The audio isn't always perfect. Some ideas are to have an easy way to let users "flag" a bad pronunciation and even correct it for other users by typing out the word phonetically, or allow user-recorded audio clips.
• In order to get it out there sooner, we launched without having a page footer or the ad unit that we need to include.

Finally, we threw in an easter egg to have fun and reward people inside Speller (hint: monster truck). Have you found it?

Want to join us? We're Hiring!

We're hiring smart engineers and UI/UX designers who have a passion for creating really excellent tools to help millions of students study. If you're interested, see this page and then send us your resume and an awesome cover letter.