Amazon Mechanical Turk Survey Strategies

I recently crafted a survey to collect data on the consumption habits of gamers. I wanted to identify different types of gamers and drill down further with future surveys targeted at specific categories of game players. To do this, I used a combination of SurveyMonkey and Amazon Mechanical Turk.

Making the Survey

The first step was creating the survey. For this, I paid for an account at SurveyMonkey. I find SurveyMonkey not only has the easiest survey creation tools but great features for analyzing the data. Alternatively, you can create surveys in Google Docs for free or inside Mechanical Turk itself using HTML.

There’s a science to creating a survey. I never took a class in demography, so I relied on some books discussing proper survey technique.

A few quick tips:

  • Always ask about what the person has done, not what they will do. For instance, instead of asking how much money they would spend on a car, ask what they spent on their last car. You will get a more accurate answer.
  • Make sure they are paying attention. Especially if your survey is long, you may be vulnerable to users clicking randomly just to get through it. This Scientific American article has a neat technique for checking if the user is taking an online survey seriously. Just ask ridiculous questions at random points. If the user answers something like “Have you ever eaten a dinosaur?” positively, throw his data out. He’s obviously not giving you valid answers.
  • Amazon Mechanical Turk

    Mechanical Turk is Amazon’s micro task site they launched in 2006. You can create tasks (called HITs) of any sort and pay workers to complete them. For instance, give someone a nickel to tell you whether a username is profane or not. In my case, I paid users 25 cents to fill out my survey.

    I used this blog as a guide on how to do it. Except instead of making the survey as a web form inside the HIT’s HTML code, I embedded my SurveyMonkey survey.

    Some tips:

  • Don’t be cheap. At first, I was trying to get away with paying 10 cents per survey. At that rate, I only got 5 or so filled out a day. When I upped the price to 25 cents, I got 30-50 a day. If you look at this confidence interval calculator, you can see it takes about 400 responses to have a decent degree of accuracy. So at 25 cents you can probably get yourself 400 responses in a week. Up your payment to a dollar and you’ll get 400 in a day.
  • Don’t get excited by your first day numbers. For workers, HITs are primarily sorted by when they were created. On the first day you create your HIT it will be at the top of the list. You’ll see a significant drop off in activity after the first day. Instead of collecting results all week it might pay off to break it up into 2 day HITs. (SurveyMonkey tracks IPs to prevent users from filling your survey out twice)
  • Give workers enough time. When designing a HIT you specify a duration. Even if your survey takes 5 minutes to complete, make the HIT last at least 30 minutes. When I shortened the window to 10 minutes I started getting complaints about it expiring before workers could complete it. Yes, you will get emails from dissatisfied workers–even over a 10 cent task.

  • Amazon’s demographics are surprisingly close for the US. I had almost an even ratio of male to female responders. Racial demographics were a little off, but not enough for me to correct them. This article provides some good insight on who is using Mechanical Turk.
  • Sign up as a Mechanical Turk worker yourself to see how it’s being used. You’ll see a lot of surveys as well as tasks like transcribing audio clips or even calling customer service phone lines and rating the operator’s performance.

    You can use MTurk surveys to collect all sorts of important data on your product or service before you even start writing code. See if you are solving a real problem for paying customers with your start-up. I’ve even heard of people surveying to test how likely users are to click on different app icon designs. The possibilities are endless and the information invaluable.

    6 thoughts on “Amazon Mechanical Turk Survey Strategies

    1. Pingback: Android Users Are Apathetic « Ralph Barbagallo's Self Indulgent Blog

    2. Thank you for sharing. This was very interesting and helpful as we’ve just started using turk for some research. Was curious how did difficult it was to have turk “speak” with survey monkey. Was much coding involved or does turk/monkey offer a simple tool for this?

      • Well they don’t have to ‘speak’ to each other really–I think one of the links in the post describes how to set it up.

        Basically you make a web form in your Turk HIT that has a the surveymonkey embed code and a text field where the responder types in the code that’s on the last page of your survey. It’s just a little HTML.

        • How does this code work? Can the worker not submit for payment if the code is wrong? How do you program MechTurk to recognize the right code? Or do they just enter a code that you then have to check on your own before approving for payment? The site offers no information on this detail.

        • Well, it’s manual. MTurk will just show you a list of workers and the code they entered. I just approve them all regardless. But if you want to approve only ones with the correct code, you can do it manually.

          You could probably use the MTurk API to only approve proper codes

    Leave a Reply

    Fill in your details below or click an icon to log in: Logo

    You are commenting using your account. Log Out /  Change )

    Google photo

    You are commenting using your Google account. Log Out /  Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )

    Connecting to %s