Running your experiment on MTurk

Introduction

In this blog post, I provide a hands-on guide on how to make your experiment available on Amazon's Mechanical Turk (MTurk). In the first part, I focus on experiments that are built in oTree and in the second part I focus on experiments that are built in Qualtrics. After reading this blog post, you will be able to make your experiment available on MTurk. Feel free to share your preferred way to running experiments on online labor markets in the comments section.

oTree

To make your oTree experiment available on MTurk, it is important that your experiment is hosted on a server. We have another Accounting Experiments blog post and two videos available on how to do this.

Once you have successfully hosted your experiment on a server, make sure to edit your ‘Config Vars’ in a way that allows you to run your experiment on MTurk. For instance, in Heroku you can select ‘Settings’ and then ‘Reveal Config Vars’. Here, you need a number of variables.

Two important variables that let oTree access your MTurk account are ‘AWS_ACCESS_KEY_ID’ and ‘AWS_SECRET_ACCESS_KEY’. You can find the values of these variables by signing in to the AWS Management Console using your Amazon AWS Account. Select your username in the top bar and select ‘My Security Credentials’. Next, select ‘Access Keys’ and ‘Create New Access Keys’. Copy both and insert them in your Config Vars.

Next, set ‘OTREE_ADMIN_PASSWORD’ to any password, ‘OTREE_PRODUCTION’ to ‘1’, and ‘OTREE_AUTH_LEVEL’ to ‘STUDY’.

Now all configuration variables have been set, you can customize how your HIT looks to participants by editing ‘mturk_template.html’ in your oTree project (_templates -> global). An example code provided by oTree is:

<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>

<crowd-form>
  <div style="padding: 20px">
    <p>
      This HIT is an academic experiment on decision making from XYZ University....
      After completing this HIT, you will receive your reward plus a bonus payment....
    </p>

    <p>After you have accepted this HIT, the URL to the study will appear here: <b><a class="otree-link">link</a></b>.
    </p>
    <p>
      On the last page, you will be given a completion code.
      Please copy/paste that code below.
    </p>

    <crowd-input name="completion_code" label="Enter your completion code here" required></crowd-input>
    <br>
  </div>
</crowd-form>

In case you want MTurk qualifications, you can edit this in oTree's settings.py. Some popular MTurk qualifications are:

mturk_hit_settings = {
  'qualification_requirements' : [        
        # Only US
        {
            'QualificationTypeId': "00000000000000000071",
            'Comparator': "EqualTo",
            'LocaleValues': [{'Country': "US"}]
        },
        # At least 500 HITs approved
        {
            'QualificationTypeId': "00000000000000000040",
            'Comparator': "GreaterThanOrEqualTo",
            'IntegerValues': [500]
        },
        # At least 95% of HITs approved
        {
            'QualificationTypeId': "000000000000000000L0",
            'Comparator': "GreaterThanOrEqualTo",
            'IntegerValues': [95]
        },
        ]
        # Masters
        {
            'QualificationTypeId': "2F1QJWKUDD8XADTFD2Q0G6UTO95ALH",
            'Comparator': "Exists",
        },
        ]
  }

The first one limits the participant pool to workers that registered as being in the United States. The second qualification makes sure that the worker had more than 500 HITs approved, and the third that more than 95% of HITs were approved. The fourth qualification makes sure that only MTurk workers are recruited that have the ‘Masters’ status.

In oTree you do not to worry as much about completion codes as in Qualtrics, as oTree automatically tracks this. However, you can use a verification code as an extra layer of verification. Most MTurk workers are used to this.

To run your experiment on MTurk, go to your oTree admin panel and select ‘Sessions’. Log in using your username and password, and click on the drop-down arrow next to ‘Create new session’. Select ‘For MTurk’, and subsequently select the session and the number of MTurk workers you want to recruit.

For development and testing you can use the MTurk Sandbox, but if you are ready to deploy your app to MTurk you can uncheck ‘Use MTurk Sandbox’ and select ‘Publish HIT’.

Once your experiment is finished, you can pay participants using the ‘Payments’ tab in the oTree admin panel, and download the data in the ‘Data’ tab.

Qualtrics

If you have built your experiment in Qualtrics, you can distribute your experiment by either using the anonymous link or the personalized link. On MTurk it is important that you can identify who completed your experiment for payment purposes. In incentivized experiments you wish to know how bonuses should be allocated among participants. Participants conduct the experiment in Qualtrics and payments occur in MTurk, so you need a link between the two. In this part, it is important that you are already registered as a requester at MTurk.

The easiest way to create such a link is to follow these steps:

1. Generate a Qualtrics Completion Code: Make sure that incoming participants are assigned a random number (code) by Qualtrics, this will be the participant's unique completion code. To generate a unique completion code, there are a few steps. First, in your survey scroll down to the bin. Just above the bin there is a message that will be shown at the end of the survey. Second, change your survey termination from default to custom and create a new message in your library called MTurk Completion Code. The body of this message should be something similar to the following:

   Thank you for completing our experiment. Your response has been recorded.
   
   Your MTurk completion code is:
   ${e://Field/MTurkCode}
   

   Next, go to the Survey Flow tab and add a new element. Choose Web Service and enter http://reporting.qualtrics.com/projects/randomNumGen.php. If you click on Test URL you will see that Qualtrics provides you with a random number. As these random numbers tend to be quite low, it is possible that two MTurk workers will receive the same random number. To address this concern, you need to click on ‘Add parameter to web service’ and add two values. A min of 0 and a max of 999999999999. In this way, it is nearly impossible that two participants will receive the same number. Change the name of the variable from ‘random’ to ‘MTurkCode’, such that the random number can be called by Qualtrics in the message at the end of the survey.

2. Create a Human Intelligence Task (HIT) on MTurk: Choose Survey -> Survey Link -> Create Project. In the next screen, you need to specify the details of your project. Once you filled in these details and move to the next screen, you are able to edit the layout of your HIT. Make sure that both the survey link and a field for participants to provide the survey code are available. After finishing the experiment in Qualtrics, participants receive the completion code and can enter this in MTurk.

3. Compare Qualtrics-assigned codes with MTurk codes: In an Excel-spreadsheet, you can compare the Qualtrics-assigned codes to the list of MTurk-entered codes. For each matching code, the payment can be approved. Based on your data in Qualtrics you can also manually allocate bonuses.

Ready to Run Your Experiment

After following the steps above, you should be able to run your experiment on MTurk. If something is unclear, there is also an Accounting Experiments video tutorial on MTurk. In case you have any questions, feel free to raise them in the comments section below.

How to reference this online article?