Latest Updates

Inside Quizlet's incredible feedback center

January 15, 2014

Today we'd like to share Quizlet's secret weapon: our custom-built feedback center.

When I started Quizlet, one of the first things I built was a small admin tool to receive and respond to feedback messages from users. I spent a lot of time talking directly to our early users, by email and sometimes by phone. Quizlet has grown a lot since then, and we now get around 20,000 feedback messages a month. We personally respond to almost every one of them, without using response templates.

Here are the highlights of our system:

  • We can pinpoint user problems very quickly thanks to a deep integration with our database and logs.
  • Our entire team uses this system as a vital part of doing their jobs, so knowledge of issues is distributed across the whole team.
  • We've built the system to make it easy to identify patterns, bugs, and fast-emerging issues.
  • We make lots of users very loyal by being human and personally responding to almost every message.

When someone writes to a company's user support with a vague message, like "the page broke," figuring out the problem can be a time-consuming back-and-forth process. But because our feedback system is deeply integrated into our site data and logs, we can often pinpoint the problem without any additional conversation. There's also no wall between our users and the people who build the products -- if someone writes in with a design issue, there's a good chance they'll get a response from our designer, Anand. Over the years we've tweaked the system hundreds of times, and made it an incredibly efficient process for our entire team to use. As a result, we get a huge amount of value from our user support messages with minimal resource drain. Here's what it looks like:

Screen Shot 2014-01-09 at 5.41.28 PM

Some brief background: Quizlet is a top 100 U.S. website, with 20 million unique users per month. Quizlet is an essential service for students and teachers around the world, and we care a lot about solving the day-to-day problems they have with us. Our feedback center has become our most important tool in identifying problems with our service, and has scaled extremely well with our growth.

Everybody does feedback

Every employee on the team has unmediated access to the system, and frequently checks it for the latest news on how people are experiencing Quizlet. Our feedback system is the homepage of our admin center, so there's no separate system to log in to. Our developers all use it to monitor for bugs and keep in touch with users. Because they use the tool frequently, they're automatically motivated to continually tweak it and make it better.

Let's take a look at how this system works. First, we want to understand our current status at a high level. At the top of the page, a row of icons in the header shows how many messages each staff member has responded to in the past week:

Feedback Heads

Because everyone is encouraged to respond to some feedback every week, the distribution of knowledge about current issues on the product is spread across everyone on the team.Β A color bar at the top shows the latest 200 messages and who responded to each one. This lets us see at a glance who's been handling recent messages and how far behind we are:

Screen Shot 2014-01-13 at 12.33.28 PM

An incoming message

When a new message comes in, we automatically categorize it by its source so we can see feedback messages from different areas of the site in one click. This means our Android team can quickly scan for problems in their app and our web teams can look at feedback on only recently launched features.

Categories

Now, let's look at each individual message. We've designed each message to show the maximum amount of information that will help us respond to the user quickly and accurately.

We show information about their device, their account, their location, their recent actions, and their history of feedback with us. This user is having problems with our Speller feature, but we also know they are in New York (geography can sometimes be the cause of an issue), using Internet Explorer 9, and they've gotten some errors logged. They've also written to us 3 times, so their previous messages might have some clues.

Message

At the top of each message, we show how many times they've used all our study modes, and if they use our iPhone or Android apps. We include information like this, along with their profile icon, to give us a more personalized understanding of who they are. A heavy user of Quizlet might look like this:

Screen Shot 2014-01-08 at 4.54.33 PM

Integrated log viewer

The most useful tool for analyzing problems is our log viewer. Each message is shown along with all of the data logs we've recorded for that user. If the user has recently seen an error message for example, we can see what that error was. We record errors from both the server side and the browser side into the same group of logs, and preview all of them inline in the feedback message. From the above message, it looks like we've recorded some javascript errors related to speller. Looks like a good clue!

Screen Shot 2014-01-09 at 8.41.15 PM

Clicking on the logs lets us see the full logged messages in our log viewer. Seeing the full messages often gives us clues about what might have gone wrong for the user. In this case, the user is getting a "Not implemented" javascript error, which seems vague, but suggests that we may be calling an undefined function in our Speller mode, which is something worth investigating.

Log Viewer

Finding patterns in feedback

One person understanding an issue is not enough; for everyone on the team to understand, we needed tools for cataloging groups of feedback and flagging actionable issues. For that, we built a system for creating "cases," where many feedbacks can be grouped together. It's useful for identifying trends and larger issues; it's also helpful for bugs we can't seem to figure out because we can group related messages together.

Any time you want to add a case to a feedback, you can click "+ Case" and it will give you an auto-completing list of current cases. For example, someone may be asking us to let them record audio on Quizlet:

Screen Shot 2014-01-08 at 4.13.38 PM

Recent active cases are displayed at the top of every page, so everyone on the team knows the pulse of current issues.

Tags

If you click on a case, you can see all the feedback messages for that case, and see the frequency trend of that problem. In this case, people have been asking for this new feature a lot recently!

Screen Shot 2014-01-08 at 3.59.58 PM

Fixing bugs

We use Github to track all the current bugs in Quizlet. If the case you're looking at is a bug that warrants being fixed, you can turn it into a Github issue in one click: Case on Github

When you've been tasked with fixing the bug, you can view all the related feedback messages and their context in one click. This makes reproducing issues and identifying patterns that cause an issue extremely easy. And when you're done, you can click "Email everyone" and send a note to everyone who was affected by the bug, letting them know it's now a thing of the past!

Generating user loyalty

Because we manage the system ourselves, we've added lots of little niceties over time. For example, in the barebones email response template for each message, we automatically add a special thank you for paid subscribers who wrote in for the first time:

Screen Shot 2014-01-09 at 2.36.01 PM

We see this tool as a primary way to generate warm fuzzies from our users. People aren't used to getting a personal, real message via online feedback forms. So we often surprise our users with the fact that they're dealing with real humans.

As Quizlet has grown, it's been a priority to maintain our deep understanding and involvement with the issues of our users. We spend a lot of time responding to feedback, but it's no one's full-time job. We have 3 people whose job description includes responding to feedback, but it's far from 50% of how they spend their time. Providing personal care for all of our users isn't easy, but the efficiency built into this tool makes it possible. Instead of seeing support as a necessary cost of running the business, we see this system as a main tool enhancing our ability to understand our business.

Many people tell us that we should package our feedback center up and sell it as a product. The main secret to its usefulness, however, is its level of custom integration with our product. So our response is usually, "You should just copy us instead!" Because we control exactly how it works, our developers can add and remove data for it in minutes. The tool has full access to all of our data and logs, letting us richly integrate any information we need. With each passing month, the tool's ability to provide context for our users' feedback messages increases.

Comments

  1. gfaout 

    First Comment.

  2. Meowstic Plus

    2nd!

  3. Dinklebird 

    3rd

  4. Dinklebird 

    highest ever been

  5. Sola6174 

    5th! Highest ever aru! :)

  6. Meowstic Plus

    Oh, and I have a suggestion. You know how in learn and space race it only counts it right if you type the word EXACTLY? Well, I think you should be able to choose some other ways that the word can be typed and it is counted correct. You see, sometimes I do words that have something in parenthesis such as Oxygen (O), and it counts it wrong if you don't type the word along with the word in parenthesis. So, basically, my suggestion is that you can type alternative ways to spell the word that it could count it right as (this would also be helpful for words that are spelled certain ways like U.P. but some people type it UP). Anyways, thanks for listening!

    -Meowstic

  7. Sola6174 

    Wow. Your feedback center is amazing aru.

  8. penguins0124 

    10th comment. Yeah!!!

  9. hi-5 

    You guys are amazing. Thank you for all you do!

  10. evan_silverman 

    11th

  11. zhou_shan 

    12th

  12. jholyoke 

    I just signed up for Quizlet. I have confirmed my email. When I log in and try to search something. It will not search.. Help!!

  13. and4131 

    14th!

  14. dillonthetruth 

    PLEASE MAKE A REAL TIME CHAT ROOM!!!!!

  15. dillonthetruth 

    INSTEAD OF THE DISCUSSION all though it is cool

  16. drunkpandas 

    ya that would actually be really cool :D haha although I don't know if that just kinda defeats the purpose to study ;P haha but still a good idea :) (dillonthetruth)

  17. Evin-jaff 

    I would like a new game to play while studying

  18. gingersnapz03 

    Yeah I agree with Evin-jaff... some new games would be AWESOMEsocks. But for now what there is is still great but new stuff is always funπŸ˜ƒ. And chat rooms would be cool too except isn't that sort of kind of like the discussions/classrooms? Idk I haven't been using Quizlet very long so still getting used to it... OMW though it helps me so much!! Thank you so much Quizlet team for working so hard! I have fun with Quizlet and use it allllll the time for studying THANK U!!! ✨ PS 19th comment!! shoutout 2 gingers πŸ˜‰

  19. gingersnapz03 

    Oops wow ginger can't count I meant 20th and 21st comment! Yeah! πŸ˜‹

  20. Nepal_is_my_username 

    hi

  21. ositan-PeacemakerPax 

    Hey, Quizlet team; do you guys actually look at the feedback that goes on down here in the comments section, or does everything need to be submitted formally?

  22. kaseyreed87 

    hi

  23. kaseyreed87 

    i live in St.Charles mi and i use Quizlet everyday because i have exams today and friday

  24. Thunderdash13 

    Feel free to join my class for good vocab cards!! It's sfxp class of 201;

  25. edithchavez 

    this is stupid

  26. jc_da_boss 

    http://quizlet.com/class/925419/

    Come and Join my classes! For awesome Dance Dance Revolution!

  27. gretald 

    30th comment!

  28. Our_Gods_not_dead 

    :)

  29. bellini01 

    32

  30. bluedomke 

    34 comment! Thats closets to first comment iv ever been!
    Wow you a lot of feed back! some of it is from me!

  31. Lit_Misfit 

    I thought that you just fixed the problems. I've never recieved a response message

  32. Assassins-Creed3 

    it wont let me hi five

  33. taylor_burns 

    I think this is really cool! I love quizlet and the way we are dealing with "real" (haha) people. :) LOVE YALL!

  34. atg6 

    38 hope it turns out well!!!!

  35. kelleywren 

    Thank you for having Chemistry "language." It is rare to find an online source that recognizes that need. I am encouraging all my students to use your site. Thank you, again.

  36. lyndsayluvs2write 

    i don't think that they look down here, actually. Maybe. And I'm curious as to why I can't find my chat log. I used to be able to track all conversations i'd had, and it was really easy to follow. Am i the only one bugged by this? And I haven't gotten an answer from Quizlet yet about that. And i've e-mailed twice. I think I just lost a screen somewhere, and it might be my computer....

  37. Raana 

    40th not bad. Very good idea.

  38. amalianc Staff

    @ositan-PeacemakerPax - We look at blog comments every day, but the best way to reach us is our feedback center - quizlet.com/feedback

  39. matthew240 

    Quizlet should allow people to chat with other people in their class

  40. tjstol2000 

    43rd comment.

  41. frogboy2013 

    45

  42. AlleyWay_SGCS 

    Woah. I personally think you should put those colored bars up on the website for everyone to see!

  43. evantgrahn 

    46th :( :( :( :( :( :()

  44. The_Mastermind Plus

    nice quiz let!

  45. The_Mastermind Plus

    :)

  46. matthew240 

    47th

  47. Tnguyen679 

    I think you should make a Tic-Tac-Toe sort of game where if the person who is playing wants to move in a certain spot they have to get the answer correct and if they get it wrong they lost their turn. It should be multiplayer, 2-person game on one computer, or against a bot.

  48. e_shan 

    wow. this is a very complex system for getting feedback that really works. i am pretty impressed.

    48th btw

  49. Tnguyen679 

    Also a space shooter game would be nice too with the same concepts as the Tic-Tac-Toe game. THANK U AND HOPE IT GETS ADDED

  50. brestelj 

    50!!!!!!!!

  51. AriStudyGroup 

    On my Quizlet wish list I wish for:

    1. Chat rooms for classes so you can teach your classes live and get feedback
    2. Class status- other people can know if they want to join the class and what is new with it.
    3. Assignments- the "teacher" can assign assignments (essays, homework, etc.) that is optional and add a caption explaining who it would help for what class. And once the class follower has finished, s/he could "turn it in" to the teacher to check and the assignment would've helped the student get prepared for any upcoming tests or if they aren't sure if they understand the subject. The teacher can check and grade it back, so the student knows how much they know.

  52. like_A_boss01 

    55
    and 56
    and 57

  53. frogboy2013 

    :roll:

  54. jasonsrvca 

    somewhere between 56-59 :D

  55. Christian_Roy 

    Thanks for sharing this. I can't help but see a parallel with something I am currently reading elsewhere... I'm reading The Cathedral and the Bazaar the Eric S. Raymond. There is a part that explains why QA in open source projects is often quicker and better than in "regular" projects. The author makes a point that open source project contributors, mostly technical people, will likely report bugs more effectively than non-technical people. Their reports are more effective because they are "closer to the code", and can typically be resolved more quickly, without back and forth conversation with the tester/reporter. I was wondering how this could apply to QA or bug management in general. It seems you figured a practical implementation of a solution with your integrated log viewer...

  56. ErzaScarlet 

    I wish there were more things that you can do to study. Somewhere between 58-61

  57. hockeystar Plus

    when you send a bug or a complaint, will you reply to our email or something else?

  58. RGGeorge 

    This is absolutely stunning. The deep integration of Quizlet user data into the feedback cycle has allowed you to do amazing things. I would give my iphone to have this kind of system.

    One question is why Quizlet doesn't harness community resolution in the way that facebook.com/help or productforums.google.com does. This can be a really meaningful way for power users to contribute to the site.

  59. asuth Staff

    RRGeorge, we've talked about doing something like that bug have never gotten around to it!

  60. asuth Staff

    hockeystar, we'll reply to your email

  61. minecrafter123456789 Plus

    HI IS IT POSSIBLE IF QUIZLET CAN ALLOW DISCUSSIONS IN CLASSES INSTEAD OF HAVING TO MAKE A DISCUSSION SET?
    IT WILL BE SO MUCH EASIER THAN MAKING SETS IN SECRET

  62. minecrafter123456789 Plus

    PLEASE POST A POST ABOUT WHETHER YOU ARE ABLE TO! THANK YOU!

  63. CrisDilworth Teacher

    Those are some nifty user stats. It would be great if users could see their own stats.

    Cris

  64. rosiethecat101 Plus

    Thanks Quizlet for everything! I love how there are always some new and neat updates and formats you make! Keep up the great job, Quizlet!

    πŸ°πŸ°πŸ°πŸ°πŸ°πŸ°πŸ°πŸ΅πŸ΅πŸ΅πŸ£πŸ£πŸ£πŸ£πŸ£πŸ£πŸΆπŸΆπŸΆπŸΆπŸΆπŸΆπŸ·πŸ·πŸ·πŸ·πŸ·πŸ·πŸ―πŸ―πŸ―πŸ―πŸ―πŸ―πŸ―πŸ˜‹πŸ˜‹πŸ˜‹πŸ˜‹πŸ˜‹πŸ˜‹πŸ˜‹πŸ˜ŽπŸ˜ŽπŸ˜ŽπŸ˜ŽπŸ˜ŽπŸ˜ŽπŸ˜ŽπŸ˜¬πŸ˜¬πŸ˜¬πŸ˜¬πŸ˜¬πŸŒŸπŸŒŸπŸŒŸπŸŒŸπŸŒŸπŸŒŸπŸŒŸπŸ‘πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘ŒπŸ‘ŒπŸ‘ŒπŸ‘ŒπŸ‘ŒπŸ‘ŒπŸ‘‹πŸ‘‹πŸ‘‹πŸ‘‹πŸ‘‹πŸ‘‹πŸ‘‹πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘πŸ‘πŸ’œπŸ’œπŸ’œπŸ’œπŸ’œπŸ’œπŸ’œπŸ’šπŸ’šπŸ’šπŸ’šπŸ’šπŸ’šπŸ’šπŸ’™πŸ’™πŸ’™πŸ’›πŸ’›πŸ’™πŸ’›πŸ’™πŸ’™πŸ’™πŸ’™πŸ’™β€οΈβ€οΈβ€οΈβ€οΈβ€οΈβ€οΈβ€οΈπŸ’—πŸ’—πŸ’—πŸ’—πŸ’—πŸ’—πŸ’—πŸ’—πŸ’žπŸ’žπŸ’žπŸ’žπŸ’žπŸ’žπŸ’žπŸ’žπŸ’–πŸ’–πŸ’–

  65. rosiethecat101 Plus

    70th? This is worthlessssss πŸ˜‚ lol

  66. Dancin4Evah 

    what's the point of this Inside Quizlet post?

  67. arpanlikuu 

    I agree with Dancin4Evah. I, at first, thought the feedback system was changed to make giving feedback more efficient.

  68. Dr-Meghana 

    This is one of the top studying websites that has helped me. Thank you very much.

  69. NinjaKirby 

    I liked how you guys started multiplayer but took it down.... I understand how
    1. It wasn't going too well
    and
    2. There's a low chance of a friend online
    But come on, we could practice with a random person around the world.

  70. Seanmt 

    Those "First!" commenters are really annoying... Who cares what number comment you have!

  71. JLee_Johnson 

    I owe your site a rating, but every time the "Rate This Site" prompt appears, it happens to be an inconvenient moment. I will try to figure out how to provide my rating at some point. (To give you an advance, it will be positive.) My real point has to do with sound / voice when using my flashcards, and this may be an issue with my i-Phone, and not your program. I use Quizlet for my Arabic vocabulary, and what I find particularly advantageous about the service is that I can create my own vocabulary lists that I make up from unfamiliar words encountered while reading the Arabic press. On my computer, I can get sound for all words. But, on my i-Phone (version 4; AT&T 3G), I cannot; sound seems to be "hit and miss," that is, it works for some words, not for others, and there is no pattern I can discern. Maybe I need to upgrade my phone, but I am asking if others have mentioned this problem, and if there might be a known fix.

  72. ChristhekidBG 

    PLS JOIN MY CLASS. I REALLY would appreciate getting 100 MEMBERS BUT I only have 57. Anyways quizlet your awesome. And my names Chris!!! Oh wait. My name is Chris. Nevermide

  73. fhaney Plus

    How can I save or transfer content information from Quizlet to another source to save it as a "backup" in case I lose Quizlet or it goes down?

  74. tktmquiz 

    another user seems to have typed in many of the same flash cards i'm typing in. i am auto defining with their info. how can i figure out where their info is, so i can copy the words they input already?

  75. deo469 

    good thing you guys got my messagge

  76. Lit_Misfit 

    wait, you guys actually SEE and ANSWER all the feedback we send?

  77. rejoy 

    HOW CAN I SENT THE EMAIL TO MY CLASS TEACHER PLZ REPLY ME BACK

    THANK YOU
    REJOY REJI

  78. Atah Plus

    Wish you could said messages to a student/teacher on Quizlet. It minimizes the time wasted to email and stuff.

  79. Gabrielspenshaw200 

    Good for you

  80. Lady_of_Lorien 

    I want to thank the entire Quizlet team sincerely for their hard work and dedication. Responding personally to each question or e-mail would is a lot of work, I know! I also appreciate that you read the comment section and do your best to help people out. You guys are phenomenal.
    Thank you from the bottom of my heart,
    Galadriel

  81. MmeThomas Teacher

    I love your new folder feature. However, I can't search other folders. It would be great to search "French Folders", "Upper School Spanish Folders" , etc. Also, if I like a teacher or student set, I'd like to see all of their sets AND their folders, if any and I'd like to be able to COMMUNICATE with them and SHARE my sets/folders, for example. On a different note, I sent an e-mail concerning taking OUT the words for "Learn Speak" so it's not so easy for the user to type the answer. They should only be able to HEAR "La maison" and type "the house" rather than SEE "la maison" and hear it at the same time. It's not strict listening comprehension if I get to see what I'm hearing. It's not challenging enough. I've also requested that the answer be spoken once it's typed in. If the answer is "un chat" and I type in "un chat", let me hear "un chat" first, rather than "a cat"---"un chat" because students hit enter too soon and miss the opportunity to wait long enough to hear "un chat". Thanks!

  82. RubikSolver 

    100th Comment?

  83. AgentDolly 

    101st comment....

    Cool! ; )

  84. viviennetang 

    102th comment! SWEET

  85. viviennetang 

    🐰🐡chick🐢pigπŸ―πŸ˜‹πŸ˜ŽπŸ˜¬πŸŒŸπŸ‘πŸ‘ŒπŸ‘‹πŸ‘πŸ’œπŸ’šπŸ’™πŸ’›β€οΈπŸ’—πŸ’žπŸ’–

  86. Miss_Independent_cat 

    104 anyway when is this comming

  87. DSMS_Runner13 

    I just want to say thanks, for all you have done Quizlet. Everyday I look at the " behind the scenes" and all the new updates, and think WOW! You all put so much effort into everything that you do. It's great to know that there is one website I can always count on, to help me study. You all should be proud of the accomplishments that you have made.πŸ˜„

    Thanks Again,
    DSMS_Runner13

  88. THEPURPLEPEOPLEEATER 

    106

  89. Wesley_Graham 

    93rd comment!!

  90. JRRT 

    94th. RubikSolver counted wrong

  91. Lady_of_Lorien 

    Thank you to all the Quizlet staff who keep this wonderful website going! To everyone: check out my quiz How Lord of the Rings Could Have Ended Quickly or refer it to someone you know who likes LOTR. If you have any other awesome ideas about how it could have ended, add them into the comments. And to the two people besides me who have already looked at it, if you added a comment, I didn't get it. Comments are only kept for one week and I didn't get to see any if I did get any...so please re-add them if you even did.Thank you!
    Signed,
    Galadriel
    P.S. 95th!

  92. RubikSolver 

    96th!

  93. xXRIGBONEXx 

    THE FEEDBACK IS NOT GOOD TRUST ME. ITS BEEN 17 MONTHS SINCE MY LAST EMAIL TO THEM. *lord*

  94. xXRIGBONEXx 

    OOOOOOOOH 98th!!!!

  95. iOS_8 

    cool

  96. iOS_8 

    cool

  97. Davidmask Teacher

    How can I get a Large Quizlet T-shirt?

  98. liamlee 

    102!

  99. lilylion678 

    103 This is kinda cool! :)

  100. AngelaJoy13 Plus

    This is incredible! Kudos kudos kudos
    You guys are amazing! Thank you for all that you do!! This is why I'm proud to be a Quizlet Plus user; I want to support you guys as much as I can :)

  101. jc_da_boss 

    YEÆ

  102. blazer46348 

    awesome and hard to understand at first
    :) lol

  103. pup424 

    106th! Record!

  104. annamccorkle 

    Wow, so cool! and @xXRIGBONEXx I have sent them emails before and i always get an answer back in about 1-2 days. ry checking your email/spam again! :)

  105. MB39 

    Hey, I have some questions!
    Questions:

    1. How do this because I am trying?
    2. Can you guys do it with classes instead?
    3. Can we have more games? (so studying could be more funnier)
    4. Are you thinking of a new game?
    5. How do make your picture moving likejc_da_boss?

Leave a comment

Login to leave a comment

Create Set