Latest Updates

How We Do Product Development at Quizlet: An Inside Look at the Making of Speller

September 20, 2011

A few weeks ago we launched our newest study mode, Speller. If you haven't used it yet, go try it. This post is more in-depth and technical than usual, but it gives a behind the scenes look at how we created Speller, our engineering challenges and processes, and how we obsessed over the user experience and the educational experience.

Summary - A Few Of The Lessons Learned (tl;dr)

  • When a technical problem is too hard, consider completely different approaches
  • Just when you think it's awesome, it could still be way better (keep iterating!)
  • Make complex user tasks (e.g., typing accents) easier by showing just what they need
  • Visualization is crucial for understanding and tracking student progress
  • Don't make it too easy; you'll lose the educational value
  • Do usability tests until recurring issues go away and it's a joy to watch
  • Loud music helps all night coding sessions!

The Beginning

It started with a simple idea: make Quizlet speak your vocabulary to help you spell. What we ended up with is a product that does that pretty well, but also is an incredibly useful tool for familiarizing yourself with vocabulary for studying a language or anything from your flashcards. Speller is so effective because it combines many "senses" at once. Let's say you're using Quizlet to study Spanish, like we are doing as a company (more on that in another post). You've got a list of Spanish Animals and you use Speller on it. You're going to, all at once:

  • Hear your Spanish terms spoken to you with correct pronunciation and a Spanish accent
  • See a text definition in English
  • See a large image representing the term (it's good to see a tree and immediately think "árbol", rather than having to translate in your head from the English word first)
  • Type the Spanish term with your attempt at its spelling
  • When you get something wrong, you'll simultaneously 1) see a visualization of what letters you got wrong and how it should have been spelled, and 2) hear the missed word spelled out letter-by-letter for you in the Spanish alphabet, followed by hearing the full word pronounced again.
  • You'll repeat the above for each term until you get it correct.

Focus on the Text-to-Speech Audio

Early last year we discovered that text-to-speech (TTS) technology had recently gotten really, really good - not just in English - but in many different languages.  After doing a bunch of research on text-to-speech engines and comparing the quality of different voices and engines in several different languages, we launched TTS in 5 languages and it was a huge success with our users. We started working hard to expand to a several of other languages, especially as French, German, and Italian were frequently requested. However, we knew that students around the world were going to be relying on this audio to help learn pronunciation, so we had a very high bar for quality. After getting native speakers from many different languages to test out the audio quality, we ended up launching 13 additional languages with TTS, giving us a total of 18.

There were some technical challenges here:

  • integrating TTS engines and voices from different companies and figuring out the technical idiosyncrasies between them
  • learning SSML
  • figuring out a way to scale to support thousands of users online at once (both technically, and from a cost/licensing perspective)
  • finely tuning encoding settings
  • getting a "spell out letter-by-letter" mode working in each language.

In the browser, we're using Scott Schiller's SoundManager2 JavaScript library as an interface for having detailed control over audio playback in every browser using either HTML5 or Flash. We learned that HTML5 audio, even in the latest browsers, is currently far from perfect, and we've been able to contribute back to SM2 both by submitting pull requests and finding/diagnosing bugs and edge cases (JSFiddle is great for this kind of thing).

A huge goal for the audio was that it needed to be FAST. That meant:

  • producing audio clips from new text snippets as fast as possible
  • getting the audio file sizes down as small as possible without losing much quality
  • utilizing various layers of caching
  • preloading audio clips so that the user doesn't has to wait to hear the next audio clip. (We setup our TTS server to add a fake 10 seconds of latency during testing, and made sure our front-end experience still felt snappy.)

Making accents easy

It's common for people trying out Speller on a foreign language to complain that it's too strict because it marks a spelling as incorrect if you simply forget an accent (e.g. typing an "a" instead of an "á"). We've decided not to be lenient on accents so that we help you learn the actual correct spellings. It's extremely common for foreign language teachers to take points off (or mark as completely wrong) answers in tests and quizzes that are missing an accent.

But we know that for the average student learning foreign languages, often just figuring out how to type "special" characters (á, ç, ü, ¿, etc.) is no easy feat. We've worked hard to make it easier. We wrote some code to look at every character in the flashcard set(s) that are being studied and determine which you might need help with. Then we show you just the buttons that you need - and no more, to keep the UI as simple as possible. (We actually wrote this feature first for Speller, but launched it back in January in the other study modes first).

As helpful as those accent buttons are, picking up your mouse is still too slow when you're going through a lot of words. So for people who were click them with the mouse, we show a little "tip" to try to teach you a keyboard-only faster way to type them.

When a problem is too hard, you might be doing it wrong

Our original prototype of Speller worked quite differently than what we ended up with. In the original, when you incorrectly spelled a term, we cleared your incorrect answer and then tried to get you to enter the correct answer by speaking one letter at a time (but without visual hints). For example, if you paused in confusion, it would say the next letter. If you typed a wrong letter, the keystroke would be rejected.

This seemed like a decent idea for a while (it simulated someone helping you spell a word) but had a number of problems, especially with the implementation. Even after a lot of time writing complicated code using all the different types of JavaScript key events, it was still pretty buggy. We could have continued debugging it, but we took a step back and realized we weren't completely in love with the idea anyways. It was kind of annoying and you ended up just pressing keys until you got it right.

Lesson Learned: Iterate & experiment a lot, but don't spend too much time on an idea that isn't "the one"

After some talking and brainstorming, we realized we wanted to visualize exactly what was wrong with our spelling. That meant we needed to show a color-coded "diff" of the letter-by-letter differences between what we typed and the correct spelling. It needed to happen client-side to be fast, and after finding (and tweaking) Google's JavaScript Diff open-source project, we had something going.

This was a big improvement in making it super obvious when you just used an "e" instead of an "a". But it didn't take long for us to realize it could still be better, and we had the idea to use an animation to "morph" your incorrect spelling into the correct one. We coded this up and loved the concept, and then we obsessed over the details of each of the different parts of the diff animation: should each part slide in from the left or right, fade in, or pop in? How fast should it slide? How much time between each part?

The last breakthrough was to not show the animation off to the right, but to literally make it look like the text you typed in the textbox is what morphs into the correct answer. With some CSS z-index and positioning trickery, the effect worked perfectly! Finally, we made sure that as soon as both the audio "spell out" and the animation diff were complete, it would move you on to try again, but you can hit any key to skip it if you're in a hurry. Here's what we came up with (excuse the video's poor audio).

When have you learned it?

By far, the most difficult part of Speller, conceptually, was the combination of figuring out a) exactly what to test you on, b) how often to test you, and c) how to present your progress on a per-term, per-round, and overall basis in a simple and concise way. We thought long and hard about how to make Speller the most educationally productive that it could be. We started with a few assertions / goals:

  1. You should spend the most time focusing on terms you're having the most trouble with.
  2. We want you to have get something right more than once before we trust that you really know it. This allows Quizlet to distinguish between you guessing the right answer once versus really knowing something.
  3. By the time you've finished, we should be pretty confident that you have really learned all the terms
  4. The overall process learning process may take a while, but we want to keep users motivated to keep going.
  5. You should be able to visualize your progress in as simple a way as possible

Checkpoint Screen

To accomplish these, we knew we want to break up long periods of studying with frequent short breaks with a "checkpoint" - giving you a chance to breathe and review what you just got right and wrong.

Attempt: "Must get 3 stars per word" Verdict: made sense; but visually cluttered end-of-round screen; was annoying to have to do easy terms 3 times; no visual progress during each term.

After some more brainstorming, we came up with a new conceptual model: You have "learned" a term once you've gotten it right twice in a row. Immediately after you get a term wrong, we'll show you the correct spelling and then force you to try again to spell it correctly before you can move on. In this "practice mode" getting it correct doesn't "count" toward getting a "correct streak of 2", but it does have the benefit of really making you take the time to learn how to spell what you missed. This seems to do a pretty good handling the first of the few goals above, but we still needed a good way to visualize the concept to the user as well as indicate their progress.

Attempt: "Show 'before' and 'after' dots representing your streak per term" Verdict: It expressed the new conceptual mode, but was way too confusing. Also, people would quickly get to the checkpoint screen but then not click "Keep Going" unless they were highly motivated. An automatic timer would help that.

Attempt: "Slide between different columns" Verdict: The sliding animation was nice and conceptually simpler than 2 sets of blue dots, but still confusing, and too much wasted screen space when you started getting a lot of terms.

Attempt: "Overall progress of blue dots, then show latest round's words and their 'dot' status below" Verdict: Getting much better! We now have a way to show the user's overall progress in learning all of the terms, as well as a review of the 7 terms they just studied in that round. But, we did some user testing and people were still confused by the dots.

Attempt: "Add labels!" Verdict: Yes! The confusion in users tests with this one went away immediately. People suddenly "got" the conceptual 2 dot model and had a quick way to visualize how their overall progress was, and what it took to "fully learn" the terms according to Speller. This is the final/launched version:

Indicating Progress Per-Term

We needed a good way to indicate per-term how you've been doing. The model of "you must get a term correct twice in a row, and getting a term correct immediately after you get it wrong doesn't count" is not particularly easy to explain, even though we believed it was best. We experimented with a dozen versions of animations and ways to indicate this in a simple way. We tried "animated brains filling up" and crazy Pacman-like ideas:

Ultimately we came up with two blue dots dropping in place or falling out of place as you get something right or wrong. It's subtle enough so that you don't have to think about it at first, but makes sense when users study with Speller long enough.

Indicating Overall Progress

We had decided on using 2 progress bars: one that moved quickly to show progress within a round (bottom), and one that moved slowly to show overall progress on how well you've learned your entire list of words (top). We started with the one of the left and found that while some people could figure out what they were for, they weren't clear enough. Tried experimenting with other ideas and finally decided simply adding labels and reducing the number of colors on the page was the best solution - and it resulted in everyone "getting it" right away.

End of Game Screen

By the end of the game, because you've had to get everything correct twice in a row, so we have a pretty high confidence that you actually know each of the terms. We wanted to give the user some helpful data so they can see their progress and a recap of the terms that they had the most trouble with. In my example below, I only got 66.4% of the prompts correct, however I still "fully learned" all my terms - it just took some time. We did some iteration on this screen as well... but once we had all of the above points decided, the final screen mostly just fell into place.

Usability tests… VERY HELPFUL

One thing we did a much better job on in creating Speller, compared to previous projects, was that we were constantly doing informal usability tests with other people. Generally we'd wait until it was at a point where we were reasonably happy, then we'd find someone to try it out. We tried hard not to cheat, meaning that we asked people to "think out loud" and verbalize questions that they had, but that we wouldn't answer them or interfere if they were confused by something. That way we got extremely useful feedback all through the process. Everyone on our team spent a lot of time helping us and trying it out, but we also paid random students in Starbucks and elsewhere around our office who were willing to spend half and hour with us to try out the latest prototype and give us their honest feedback.

What's interesting about doing testing with real users is that everyone has slightly different pain points / things they are confused by. Many of those are worth fixing, some are okay to write off. But the biggest problems are going to be recurring themes of confusion from different users, and that's where you should keep working the hardest. Usability testing can be painful - sometimes there will literally be a button right in front of their face but they won't see it. After even just a short session we'd come back with a full page a notes we scribbled about little things we noticed that could have been better about their experience.

If I had to sum up our process for creating Speller, it would be: we basically kept iterating until user testing was a joy to watch. We knew it was finally good when we'd find a random person, put them in front of Speller with a set of terms they didn't know, and have them go through the whole thing with very visible progression of learning, and no real pain points or confusion along the way.

Engineering processes & details

The overall engineering processes at Quizlet could use it's own post, but a few specifics on Speller:

  • Andrew and I (Phil) built Speller start to finish over several weeks. It was an entirely iterative process happening in code - there was no "spec" stage and the product was not fully designed ahead of time.
  • On some features we agreed the whole time. On others we started with pretty different ideas about how something should work, but kept iterating and taking the best of each idea until we agreed on a solution that was usually better than either of us started with.
  • The rest of our team was incredibly helpful: they tried out various prototypes and gave tons of great feedback and ideas. Often this feedback was that something was unclear, and it resulted in us hitting the drawing board on various concepts to come up with something better.
  • Once we felt the core flow of Speller was conceptually good, we started using GitHub Issues with a "Launch Speller" milestone heavily to track a long list of features, bug fixes, and areas for improvements that pretty much summed up the remaining work to do. Then it was just a matter of crossing them off (automatically with git commit messages) and although for a while, we seemed to add 2 new issues for every 1 we crossed off, we were able to complete all the essentials to launch. Having discrete Issues allowed us to keep track of discussions for specific features and link them to any relevant code.
  • Started using Jasmine to do some JavaScript unit testing
  • There are almost no images in Speller. We're using CSS3 and JavaScript for most of the nice visual effects. In order to create a good fluid layout, we used the CSS3 box-sizing property to abandon the traditional box model and also got images to auto-resize based on the available width
  • Painstaking testing in IE 7, 8, 9 (we have dropped support for IE6) with an attitude of progressive enhancement. Things don't have to be as nice in IE8 as they are in Chrome 14, but it should still be pretty good and fully functional. We did several passes of IE testing and fixing (didn't leave it entirely to the end), but also did big stretches of development without worrying about IE.
  • Tried to make Speller fail gracefully: code to detect when some internal state wasn't right (persisting/resuming JS game state when the underlying flashcard data could change is tricky) and showed a nice popup helping the user with the problem.
  • We use Google Analytics Event Tracking to observe how far along people get within a game
  • Speller consists of approximately 4,000 lines of code - mostly JavaScript.

Launched, but never a final product

The final week of working on Speller was a crazy one! After several nights staying up all night long (blasting Pandora to keep us pumped up), we finally launched at 7am one morning. Later that same day we had the privilege to visit a school where students use Quizlet and got to watch a classroom full of students using Quizlet and observed them trying out Speller for the first time. Though we launched Speller and think it's great, we want to always be looking for opportunities to improve and iterate. A few examples:

  • Speller is a lot like Learn mode at a fundamental level. We need to pull some of the best parts about Speller into Learn Mode, and even think hard about how we could merge the two study modes into one über useful way to study. (Note: we started by added a "Don't Speak" mode in Speller for people who love the experience of Speller, but want the difficulty of Learn Mode.)
  • The audio isn't always perfect. Some ideas are to have an easy way to let users "flag" a bad pronunciation and even correct it for other users by typing out the word phonetically, or allow user-recorded audio clips.
  • In order to get it out there sooner, we launched without having a page footer or the ad unit that we need to include.

Finally, we threw in an easter egg to have fun and reward people inside Speller (hint: monster truck). Have you found it?

Want to join us? We're Hiring!

We're hiring smart engineers and UI/UX designers who have a passion for creating really excellent tools to help millions of students study. If you're interested, see this page and then send us your resume and an awesome cover letter.

Comments

  1. Max

    Awesome work as always you guys! From a dev standpoint I am in awe (as always) with the work that you do, and as a student I am eternally grateful. I think Speller is a great new tool especially when learning foreign languages. I don't think a lot of people realize how much work goes in to developing new features like this. I think a lot do, but a lot don't. I hope as you add in more features and more ways to learn you'll keep making blog posts like these. I found it really informative, although I'm still trying to find that monster truck easter egg.

  2. Mme Thomas

    I have so much love for speller, that's going to be hard! I love how quickly the students learn how strict, yet fair "Speller" can be, with a playful touch of bouncy balls and a special reward video clip for those who are hard-core spellers. A phrase has to be mastered perfectly with spelling and punctuation twice in order to earn mastery. If there is a mistake, the mastery must start again and there is even a "just for practice" feature that doesn't count towards the mastery. I also love the break of 7 and the result bars at the end. The ultimate gift is the video at the end when one achieves a perfect 100% accuracy. I give my students "props" when they see the video, but make it very clear, it's all about completion and not the accuracy grade, because in the end, all were mastered. The students love the double bouncy ball of mastery.

  3. David

    Fantastic article... thanks for the great insight into the brilliant thinking that goes on behind the scene.

    I'm sure someone may have already thought about it, but I just wanted to suggest a fill-in-the-blank mode which i find useful in memorizing short phrases and sentences and could also be used to memorize spelling of individual words too...

    Since I study Korean, a character based language which treats groups of letters as one phonetic unit, I really need help memorizing the order of these characters when pronouncing words and repeating sentences...

    It takes a long time to copy and paste sentences and create blanks for each character or word, so I was hoping that you guys could quickly come up with code to automate this process.

    I have pasted the link to two sets that demonstrate the process. I always use it in Learn mode with Prompt with term and Stuff in parentheses selected.

    Fill-in-the-blank (1 character): http://quizlet.com/6543831/learn/
    Fill-in-the-blank (1 word): http://quizlet.com/6620868/learn/

    Check it out and let me know what you think.

    Happy coding ;~)

    P.S. - I loved this article so much I actually became a paying member today.
    Cheers and continued success to your stellar organization.

  4. 3rd!!

    speller isnt my favorite but appreciate u doing it

  5. 4th!

    i want the multilayer soon!

  6. NUMBA FOUR

    I like speller because it helps with my latin tests! THANKS QUIZLET!!!!

  7. abt79

    dang my computer has no sound

  8. milkncookies

    speller is great!!:D:)

  9. the guy who says the 7th comment

    Speller is alright, but in Speller it doesn't like typos which means i get 1 question wrong D:

  10. Speller focuses on helping you with the spelling, of course it doesn't like typos

  11. 9. 9. 9.

    i wish ud sticked with the pac man thing. I liked it!

  12. David (Dah-veed)

    Thanks you guys! I appreciate you making a program like this. Works great for learning how to spell things, especially for foreign languages. Keep up the good work! I'm going to go get the double bouncy ball of mastery now, see you later!

  13. Anonymouse

    I am at awe on the amount of high level Javascript you used here

  14. Panther

    Dudes this is amazing! BTW the video is pretty awesome 2.

  15. abcdefghijklmnopqrstuvwxyz

    i guess its okay...quizlet should make more games

  16. Cindy

    My students love the new Speller! Thanks for making it!

  17. Arkajit

    Awesome! Just tried speller on Spanish adjectives. Loved it -- the persistent iterations definitely paid off. Very effective too - a few times the audio wouldn't play, but I still remembered the word so was able to spell it from memory anyway. Especially liked the 7 word break and the diff animation. Don't know if it was a conscious decision to choose 7, but it works out well since that's about how many chunks we can store in short-term memory. Keep up the good work!

  18. Phil Freo

    @Arakjit... yep, it was :)

  19. me12166

    Wow! I love speller it's so fun to use and it's really helped me with my spelling on my spanish since I get that wrong a lot! Keep up the good work!

  20. Latin...Latin...and...more Latin

    AWESOME!!!! Such a complex complex process for a simple idea!!! Now how about Latin for Quizlet????

  21. latin wold be so helpful
    i take it in school and i can never get the accents!

  22. Jacob

    You should make a mode that allows one to use it with un-speakable languages (Chemistry?), so we can practice writing out formulas and such.

  23. Jacob

    I meant to write "...that allows one to use speller..."

  24. :)

    25th comment

  25. Anonymous

    Thanks... I love speller!!!!
    Monster Truck is Awesome!

  26. Anonymous

    Also please make more games!!!!

  27. so_sew

    I found the easter egg!!! I like speller in that it test you twice, but I am doing medical terminology and it can't really pronounce the words properly. Sometimes if it can't say the word it just spells the word. I do like the don't speak option though.I love quizlet because of the different games and options you get for entering just one set of data.

  28. ?

    great i discovered was trying to find something to study for french last minute and i found quizlet's speller and it helped a lot

  29. Gerard

    Great job! Really useful and helping me a lot with my Brazilian Portuguese learning.

  30. Eddie Schodowski

    I could not be more happy with Quizlet. I hope one day typical high school students understand - it's fun to learn!

  31. Jim Damblimboroski

    HAHAHA! WHAT KIND OF LAST NAME IS "SCHODOWSKI"? STOOOPID. HAHAHAHAHA

  32. CinnamonRolls17

    Love ya Quizlet!

  33. SkyDays

    Speller is awesome!! Keep makin' it better!

  34. Euan

    Hi - characters and symbols are auto-detected... how to remove them?

  35. orangec2003 Plus

    35th comment!!!!

Leave a comment

Login to leave a comment

Create Set