How to set up a simple experiment in Qualtrics

Mar 17, 2021

Many doctoral students and researchers interested in conducting experiments struggle to find a place to start. For these people, Qualtrics is well-suited because it offers a simple point-and-click interface to design experiments. In this post, I show two ways to use Qualtrics for experimental research. I also discuss features Qualtrics offers to ensure participants are paying attention during your experiments.

A great place to start learning about how to conduct experiments in Qualtrics (or any other platform or programming language) is always the documentation page. For the purpose of this post, I rely on an example Qualtrics project, which can be downloaded here. The file must be downloaded and imported as a Qualtrics project. The example is based on the instrument used in Ghita (2021). In this experiment, I manipulate the frequency with which employee performance is reported to supervisors. I recruited participants using Prolific.ac.

The fast way to create experiments

To create an experiment fast you need to follow three steps:

Randomly assign participants to different conditions
Show different information to participants based on their assigned condition
Measure the effect of the information on the dependent variable

The fastest way to implement this in Qualtrics is to show different Blocks to participants depending on the value of an Embedded Data Element:

Step 1 - Randomly assign participants to different conditions

In Survey Flow view, randomly assign participants to conditions by using a randomizer to set the value of an Embedded Data Element (let's call it Condition). The embedded data will show up in the results and can be used to present different information to different participants.

Step 2 - Create information blocks for each condition

Create a separate Block for each condition that contains the information that participants need to see in each condition. For example, in the example experiment the LowFrequency Block contains the following information:

Regional managers have full discretion over the bonuses of the salespeople in their region and can use any information at their disposal when evaluating the salespeople. To aid with the bonus decisions, the accounting department of CoffeeAndGo provides regional managers with reports containing the profit figures realized by every salesperson at the end of every six months.

While the HighFrequency Block contains the following information:

Regional managers have full discretion over the bonuses of the salespeople in their region and can use any information at their disposal when evaluating the salespeople. To aid with the bonus decisions, the accounting department of CoffeeAndGo provides regional managers with reports containing the profit figures realized by every salesperson at the end of every month.

Step 3 - Show different information to participants based on their assigned condition

In Survey Flow view, add Branches that check the value of the Condition Embedded Data and display the relevant Block for each condition.

The flexible way to create experiments

The advantage of the previous approach is that it is fast to implement. However, I prefer a slightly different approach that, although a little more complex, provides more flexibility. Notice that, in the previous approach, there is a lot of repetition between the LowFrequency and HighFrequency Blocks. This is a problem because if I want to change a detail like the name of the company, I will need to remember to change the text in both blocks. Therefore, each change adds a small chance that I will make a very costly mistake.

You can avoid this problem by showing all participants the same block in which you manipulate only the necessary words. To do this, you first randomly assign the value of the Condition Embedded Data by replicating the first step of the Fast Way. Then, in Survey Flow view, add Branches that check the value of the Condition Embedded Data and assign the text you want to manipulate to an Embedded Data Element called ManipulatedText. You can add multiple Embedded Data Elements in each branch in case you manipulate your text in more than one place. Finally, add the ManipulatedText Embedded Data Element where you need to present different information to participants in different conditions. The example from the Fast Way looks like this:

Regional managers have full discretion over the bonuses of the salespeople in their region and can use any information at their disposal when evaluating the salespeople. To aid with the bonus decisions, the accounting department of CoffeeAndGo provides regional managers with reports containing the profit figures realized by every salesperson at the end of every ${e://Field/ManipulatedText}.

Notice that this approach allows me to change any detail about the text without worrying that I will need to make the same change in multiple blocks.

Add attention checks (optional)

If you recruit participants using a Micro Task platform (e.g. Prolific.ac or Mturk) you may wish to check if participants are paying attention to your instructions. For this, I prefer to use attention questions that check if participants have understood the context of the case. I only collect responses and compensate participants who correctly answer all these attention questions. To implement this in Qualtrics:

Create a new Block that will contain your test questions (let's call it Quiz Block).
Add Score to the quiz questions.
In Survey Flow view, after the Quiz Block, set an Embedded Data Element (let's call it Score) to be equal the score variable.
In Survey Flow view, add a Branch that checks if Score is below a certain value.
End the Survey for participants who have a lower score than needed.

In the example experiment, participants can attempt to pass the quiz twice. Since even attentive participants can misclick or misunderstand a question, I think allowing participants to try multiple times is better. Allowing multiple attempts does not seem to decrease data quality. I did not find any systematic differences in responses between participants who passed the quiz on the first attempt as compared to participants who passed the quiz on the second attempt.

References

Ghita, R. (2021). Reporting Frequency and Learning from Experience.

How to reference this online article?

Ghita, R. (2021, March 17). Setting up a simple experiment in Qualtrics. Accounting Experiments, Retrieved from: https://www.accountingexperiments.com/post/simple-qualtrics-experiment/