Amazon Mechanical Turk Survey Strategies

I recently crafted a survey to collect data on the consumption habits of gamers. I wanted to identify different types of gamers and drill down further with future surveys targeted at specific categories of game players. To do this, I used a combination of SurveyMonkey and Amazon Mechanical Turk.

Making the Survey

The first step was creating the survey. For this, I paid for an account at SurveyMonkey. I find SurveyMonkey not only has the easiest survey creation tools but great features for analyzing the data. Alternatively, you can create surveys in Google Docs for free or inside Mechanical Turk itself using HTML.

There’s a science to creating a survey. I never took a class in demography, so I relied on some books discussing proper survey technique.

A few quick tips:

  • Always ask about what the person has done, not what they will do. For instance, instead of asking how much money they would spend on a car, ask what they spent on their last car. You will get a more accurate answer.
  • Make sure they are paying attention. Especially if your survey is long, you may be vulnerable to users clicking randomly just to get through it. This Scientific American article has a neat technique for checking if the user is taking an online survey seriously. Just ask ridiculous questions at random points. If the user answers something like “Have you ever eaten a dinosaur?” positively, throw his data out. He’s obviously not giving you valid answers.
  • Amazon Mechanical Turk

    Mechanical Turk is Amazon’s micro task site they launched in 2006. You can create tasks (called HITs) of any sort and pay workers to complete them. For instance, give someone a nickel to tell you whether a username is profane or not. In my case, I paid users 25 cents to fill out my survey.

    I used this blog as a guide on how to do it. Except instead of making the survey as a web form inside the HIT’s HTML code, I embedded my SurveyMonkey survey.

    Some tips:

  • Don’t be cheap. At first, I was trying to get away with paying 10 cents per survey. At that rate, I only got 5 or so filled out a day. When I upped the price to 25 cents, I got 30-50 a day. If you look at this confidence interval calculator, you can see it takes about 400 responses to have a decent degree of accuracy. So at 25 cents you can probably get yourself 400 responses in a week. Up your payment to a dollar and you’ll get 400 in a day.
  • Don’t get excited by your first day numbers. For workers, HITs are primarily sorted by when they were created. On the first day you create your HIT it will be at the top of the list. You’ll see a significant drop off in activity after the first day. Instead of collecting results all week it might pay off to break it up into 2 day HITs. (SurveyMonkey tracks IPs to prevent users from filling your survey out twice)
  • Give workers enough time. When designing a HIT you specify a duration. Even if your survey takes 5 minutes to complete, make the HIT last at least 30 minutes. When I shortened the window to 10 minutes I started getting complaints about it expiring before workers could complete it. Yes, you will get emails from dissatisfied workers–even over a 10 cent task.

  • Amazon’s demographics are surprisingly close for the US. I had almost an even ratio of male to female responders. Racial demographics were a little off, but not enough for me to correct them. This article provides some good insight on who is using Mechanical Turk.
  • Sign up as a Mechanical Turk worker yourself to see how it’s being used. You’ll see a lot of surveys as well as tasks like transcribing audio clips or even calling customer service phone lines and rating the operator’s performance.

    You can use MTurk surveys to collect all sorts of important data on your product or service before you even start writing code. See if you are solving a real problem for paying customers with your start-up. I’ve even heard of people surveying to test how likely users are to click on different app icon designs. The possibilities are endless and the information invaluable.