Gaming Coursera Quizzes

I have signed on for a number of Coursera courses now, and each of them have assigned a heavy weighting to graded quizzes. Unfortunately these quizzes are typically easy to game: it is typically possible to score 100% without knowing anything about the subject. This post shows how that can be done, and suggests a solution.

Coursera Grading

Most of the courses I have signed on for have a weekly quiz, consisting of ten questions, designed to test comprehension of that week’s material. Most of the questions are multiple choice, although there are sometimes questions that allow free text in the answers. These questions are presumably marked with natural language processing tools.

The multiple choice questions are typically presented with radio button controls, so only one answer can be selected. There are four options to choose from, so a user has a 25% chance of getting a question right if they just guess. Sometimes there are questions allowing multiple answers to be selected, but these are much less common in my experience.

Importantly, most of the courses I have looked at have allowed four attempts at each weekly quiz before a mark is committed. After each attempt, you are told your overall mark for that attempt, and then shown which questions you got right, and which questions you got wrong. So if you attempt the quiz again, your score should be at least as high as it was in the previous attempt.

The Coursesra grading system also stores more than four possible answers to each quiz question. If you attempt a question twice, you will probably see five or six possible answers over those two attempts, rather than just four.

The Hack

If there were only four possible answers to each question, and you had four attempts at every quiz, it would always be possible to get 100%, since you could simply try every possibility.

It’s worse than that, though: having more than four possible answers actually makes it easier to guess the correct answer in less than four attempts. In any series of attempts at the same question, the right answer must be one of the options every time, but wrong answers do not need to be present every time. So if you can see all of the possible answers presented each time, you can immediately rule out any possible answer that was not present in every attempt at the question.

For example, imagine you took a random guess at one question, and got it wrong. Now in the second attempt at the question, you will not try the same answer again. Which one should you try? If all of the other possible answers from your first attempt are still there, then you have a one-in-three chance of guessing correctly. What if some of the possible answers are new? That’s great, because a new answer cannot be the right one. If there are two new answers, and two repeated ones, then your chances of guessing correctly are now one-in-two.

So if you track all of the possible answers that have been shown to you, you will certainly be able to guess correctly in four attempts, but you may be able to eliminate possible choices and guess correctly in fewer attempts.

Possible Solutions

What would I do if I were Coursera? Coursera is a for-profit organisation. Although they may simply want as many students as possible right now,  I will assume they want to maintain some credibility for their courses.

If they want to keep multiple-choice quizzes as a major part of the grading, then there are three things they can do relatively quickly:

  1. Reduce the number of attempts that a student can make at one quiz.
  2. Increase the number of potential answers shown for each question.
  3. Stop choosing wrong answers for display from a larger pool.

There are two other possibilities I can see that are worth investigating, but may be harder to implement:

  1. Use multiple choice questions with multiple correct answers as well as multiple wrong answers. Obviously most questions will have to be re-written to try this.
  2. Try to detect methodical guessing by the pattern of answering, and penalise it. I’m not sure how easy that would be.
Advertisements
Post a comment or leave a trackback: Trackback URL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: