Farah Arshad | Accounting Experiments

Why We Ignore Free-form Communication and Why We Shouldn't

Mon, 07 Feb 2022 00:00:00 +0000

When someone says: “I heard it through the grapevine”, what comes to your mind first? While you may not immediately think of the typical conversation with one of your coworkers at the coffee machine, almost 70% of all communication within organizations disseminates “through the grapevine”, a.k.a. informal communication (Wroblewski, 2018). Yes, this includes the gossip train your secretary is running at your department. But it also includes giving informal advice or feedback to a coworker, reinforcing social norms, building trust, and sharing knowledge.

Communication does not stop here. The remaining 30% of organizational communication happens through more official or formal channels, such as during departmental meetings, client visits, feedback systems, and so on. Such formal settings might be more task-oriented and hierarchical, but there is at least one thing they have in common with informal communication: they both entail free-form communication.

In experimental settings, free-form communication can be defined as communication that is unstructured, real-time, and that has no or limited restriction placed on the communication by the experimenter. This could be an interaction via e-mail or chat, but it also includes audio and face-to-face interactions. Free-form communication can also involve numerous participants simultaneously.

In many accounting experiments, regardless of whether they accommodate formal or informal communication, free-form interaction is often scaled down to a restricted communication setting. However, the overarching discussion is whether research questions in accounting necessitate more complex forms of communication. In this blog, we try to offer more guidance on why and when free-form communication can be helpful in accounting experiments.

Restricted communication

There is no doubt that free-form communication is insightful and, to some degree, reflective of what happens in practice. Still, most experiments in accounting stick to a more restricted form of communication. We believe that a large set of honesty and budgeting experiments offer a fair example of this because they allow participants to send a cost report in the form of a number (e.g., Evans, Hannan, Krishnan, & Moser, 2001; Hannan, Rankin, & Towry, 2010; Maas & Van Rinsum, 2013). Similar to listening to your favorite podcast on Spotify, these experiments accommodate one-sided communication and do not allow the receiver of the report to respond to it.

Other experiments do incorporate a bilateral feature that allows the other party to respond (e.g., Chow Hwang, Liao, & Wu, 1998; Fisher, Maines, Peffer, & Sprinkle, 2002; Majors, 2016), but participants remain restricted in the responses they can provide. Participants can respond either with a numeric value or with a pre-determined written message designed by the experimenter.

While the experimental designs mentioned above might not offer a full-blown representation of reality, the restricted communication context carries several benefits.

First, there are three important principles in an experimenter’s life: safeguard the random assignment of treatments, ensure that the experiment replicates, and maintain experimental control at all costs. These principles are in fact so well-ingrained in us, experimental researchers, that it almost makes you want to abide by these rules in your daily life too. It goes without saying that we do not disregard these principles. As Steven Kachelmeier (2018) rightly said in his commentary: “Experiments are well-suited to test theories, not to simulate reality.” With this in mind, restricted communication ties in with the third principle and generally allows us to test the pre-selected theory while exercising tight control over the experiment and subsequent analysis.
Another closely related reason is that pre-defined signals (e.g., a number or pre-formulated message) are often better suited to disentangle mechanisms than free-form communication. Consider the example of budget reporting settings again (e.g., Evans et al., 2001), where the main interest commonly lies in honesty as a psychological mechanism. In such settings, honesty is measured by computing the difference between the budgeted cost and actual cost (i.e., slack), which shows that restricting the communication content to numerical values is sufficiently reflective of the underlying mechanism. If one would allow for free-form communication instead, it could introduce a myriad of social preferences other than honesty.

Garofalo and Rott (2018) provide another insightful study where restricted communication facilitates the disentangling of mechanisms. They investigate whether individuals react differently depending on (1) who communicates bad news or unfair decisions (source) and (2) how that unfair decision is communicated (style). If the researchers had not opted for pre-formulated messages in their experimental design, it would have been unclear whether the communication source or style was driving their results.

Free-form communication

In an earlier post, Jeremy Bentley (2021) correctly explained that a proper experimental design depends on both the participant pool and the research question. Surprise! Because it is no different this time. Juggling the benefits of a restricted communication setting with the benefits of a free-form communication setting also depends, by and large, on the research question. In fact, a variety of research questions, such as those described below, accommodate the use of free-from communication.

The first type of research question that could benefit from the richness of free-form communication relates to the transfer of (private) information, such as sharing knowledge or convincing someone to make a certain decision. Cheap talk offers a good example (e.g., Zhang, 2008; Lundquist et al., 2009; Lunawat, Waymire, & Xin, 2021), where communication between players does not directly affect payoffs. Restricted cheap talk experiments often present a message space that is too limited because it only includes scripted intentions (Dugar & Shahriar, 2018). The authors show that cheap talk is a lot more effective if people can explain why they choose a specific action. It even happens in daily life. For instance, imagine you and your partner are deciding to go to McDonald’s (MD) or Burger King. You suggest choosing MD because you are lazy, and MD is closer to your house. Assuming both restaurants are of equal quality (up for debate), just saying you intend to choose MD would not make the talk to your partner truly cheap and make it less likely to convince him/her that MD is the better option.

The second type of research question that allows the integration of free-form communication pertains to the holistic nature of certain types of interactions, such as those between auditors and their clients or between feedback seekers and givers. If you were present at the 2017 AOS conference, you likely remember Steven Kachelmeier’s memorable presentation, to say the least. We would like to pat him on the back (quite literally) for having the courage to present his slides with his back turned to the audience and for offering us such a fitting illustration of the value of free-form communication. As Kachelmeier argues: “The ceteris paribus nature of their [ref. Bennett & Hatfield, 2018] experiment limits the extent to which their study can address something as rich and complex as human communication.” At the risk of some self-promotion, our own research shows that settings where employees seek real-time feedback from a supervisor (Dierynck & Masselink, 2020) or where calibration committees discuss employee performance (Arshad, Cardinaels, & Dierynck, 2020) are no different.

Why is free-form communication worthwhile?

Despite the nature of the research question, there are also more generic reasons why free-form communication might be useful, and we try to outline some of these reasons below.

We may obtain different experimental results when restricting communication to numerical values or prefabricated messages. Just a brief reminder: we are not (yet) robots. We are programmed differently when we can communicate whatever is on our mind than when our communication is limited. Relative to restricted communication, free-form communication does not only enhance trustworthiness and trusting (Ben-Ner, Putterman, & Ren, 2011), cooperation in repeated games (Cooper & Kuhn, 2014), and the efficiency of promises (Charness & Dufwenberg, 2006, 2010), but it can also reduce lying (Lundquist et al., 2009).
We gain mundane realism (i.e., perceived similarity to practice). To elicit the right beliefs, more realistic scenario settings can be specifically valuable in experiments that use more sophisticated participants, such as auditors or corporate directors (Bloomfield, Nelson, & Soltes, 2016). For instance, Bowlin, Hobson, & Piercey (2015) manipulate auditors’ access to an unstructured chat with a manager to investigate whether engaging in such interpersonal chat increases the rate of low-effort audits. It is important to note, however, that there is a tradeoff between mundane realism and internal validity where more mundane realism may come at the cost of internal validity. In this case, theory-testing experimentalists generally seem to favor internal validity, but there could be ways to overcome this issue (more on this in another blog post).
We can collect richer data in a controlled environment. Especially if the experimenter’s main interest does not lie in a specific aspect of the communication process, collecting data on the communication content, social preferences or other social cues can be valuable. For instance, by quantifying the dialogues using textual analysis software or independent coding, you can shed more light on the mechanism driving the results (e.g., Zhang, 2008; Bowlin et al., 2015; Lunawat et al., 2021).

Given the above reasons, would you feel comfortable with incorporating free-form communication in an accounting experiment? In our experience, the receptiveness of the (management) accounting community to free-form communication is maturing. However, as we discuss in this blog post, a lot of times it all boils down to your research question. No method will be superior in all aspects.

So, whenever you are wearily waiting for your Monday morning coffee at the coffee machine, try to engage in a free-form interaction with your coworker, because real-life communication really is a lot more unique than the average prefabricated “Hi, my reported cost is equal to 5.00 Lira per unit” message. As George Bernard Shaw rightly said: “The single biggest problem in communication is the illusion that it has taken place.”

References

Arshad, F., Cardinaels, E., and Dierynck, B. 2020. Facing a calibration committee: The impact on costly information collection and subjective performance evaluation. Available at: https://dx.doi.org/10.2139/ssrn.3021683.

Ben-Ner, A., Putterman, L., & Ren, T. 2011. Lavish returns on cheap talk: Two-way communication in trust games. The Journal of Socio-Economics, 40(1): 1-13.

Bennett, G. B., & Hatfield, R. C. 2018. Staff auditors' proclivity for computer-mediated communication with clients and its effect on skeptical behavior. Accounting, Organizations and Society, 68: 42-57.

Bentley, J.W., (2021, July 18). Choosing the right participants for your experiment, Accounting Experiments, Available at: https://www.accountingexperiments.com/post/choosing-the-right-participants/

Bloomfield, R., Nelson, M. W., & Soltes, E. 2016. Gathering data for archival, field, survey, and experimental accounting research. Journal of Accounting Research, 54(2): 341-395.

Bowlin, K. O., Hobson, J. L., & Piercey, M. D. 2015. The effects of auditor rotation, professional skepticism, and interactions with managers on audit quality. The Accounting Review, 90(4): 1363-1393.

Charness, G., & Dufwenberg, M. 2006. Promises and partnership. Econometrica, 74(6): 1579-1601.

Charness, G., & Dufwenberg, M. 2010. Bare promises: An experiment. Economics Letters, 107(2): 281-283.

Chow, C. W., Hwang, R. N. C., Liao, W., & Wu, A. (1998). National culture and subordinates' upward communication of private information. The International Journal of Accounting, 33(3): 293-311.

Cooper, D. J., & Kühn, K. U. 2014. Communication, renegotiation, and the scope for collusion. American Economic Journal: Microeconomics, 6(2): 247-78.

Dierynck, B., & Masselink, C. 2020. Demand-driven feedback systems and employee creativity. Available at: https://ssrn.com/abstract=3738177.

Dugar, S., & Shahriar, Q. 2018. Restricted and free-form cheap-talk and the scope for efficient coordination. Games and Economic Behavior, 109: 294-310.

Evans III, J. H., Hannan, R. L., Krishnan, R., & Moser, D. V. 2001. Honesty in managerial reporting. The Accounting Review, 76(4): 537-559.

Fisher, J. G., Maines, L. A., Peffer, S. A., & Sprinkle, G. B. 2002. Using budgets for performance evaluation: Effects of resource allocation and horizontal information asymmetry on budget proposals, budget slack, and performance. The Accounting Review, 77(4): 847-865.

Garofalo, O., & Rott, C. 2018. Shifting blame? Experimental evidence of delegating communication. Management Science, 64(8): 3911-3925.

Hannan, R. L., Rankin, F. W., & Towry, K. L. 2010. Flattening the organization: The effect of organizational reporting structure on budgeting effectiveness. Review of Accounting Studies, 15(3): 503-536.

Kachelmeier, S. J. 2018. Testing auditor-client interactions without letting auditors and clients fully interact: Comments on Bennett and Hatfield (2018). Accounting, Organizations and Society, 68: 58-62.

Lunawat, R., Waymire, G., & Xin, B. 2021. Do verified earnings reports increase investment? Contemporary Accounting Research, 38(2): 1368-1394.

Lundquist, T., Ellingsen, T., Gribbe, E., & Johannesson, M. 2009. The aversion to lying. Journal of Economic Behavior & Organization, 70(1-2): 81-92.

Maas, V. S., & Van Rinsum, M. 2013. How control system design influences performance misreporting. Journal of Accounting Research, 51(5): 1159-1186.

Majors, T. M. 2016. The interaction of communicating measurement uncertainty and the dark triad on managers' reporting decisions. The Accounting Review, 91(3): 973-992.

Wroblewski, M. T. 2018. The importance of the grapevine in internal business communications. Available at: https://smallbusiness.chron.com/importance-grapevine-internal-business-communications-429.html.

Zhang, Y. 2008. The effects of perceived fairness and communication on honesty and collusion in a multi-agent setting. The Accounting Review, 83(4): 1125-1146.

Why PEQs do not provide the best process evidence

Fri, 23 Apr 2021 00:00:00 +0000

The questionnaire (or self-administered survey) was invented by Sir Francis Galton, A British anthropologist, explorer, and statistician in the late 1800s. He used them as a cheap and standardized way to elicit people’s opinions. After an experiment, researchers often also distribute a questionnaire. Traditionally, the primary role of this post-experimental questionnaire (PEQ) has been an ancillary one. They have been used to elicit a handful of stable variables that help safeguard the quality of the data and verify randomization across treatments. The variables elicited by PEQs include manipulation checks, which are statements that help evaluate the degree to which the experimental stimuli worked, and demographic variables, which help test the extent to which treatments have been randomly allocated to participants.

However, in the experimental accounting literature, PEQs serve a much more critical role because many also use them to obtain process evidence. Process evidence is empirical evidence supporting, as narrowly and tightly as possible, the theoretical mechanism central to a hypothesis. Accounting researchers often require such evidence before accepting that the results generated by an experiment support a hypothesis. Although there are many ways that controlled experiments can obtain process evidence (see, for instance, Asay, Guggenmos, Kadous, Koonce, and Libby (2021) for an insightful discussion), PEQs, by far, seem the most popular way for experimental accounting researchers to try and obtain it.

There are two broad ways accounting researchers use PEQs to obtain process evidence for controlled experiments. First, PEQs may ask participants to reflect on their behavior and decisions during the experiment they just took part in. If participants’ reflections match the maintained theoretical explanation and explain part of the observed relationship predicted by the hypothesis, experimental researchers have produced process evidence. Second, PEQs may also ask participants to reflect on their personality and general attitudes in daily life. If their personality or general attitudes change (part of) the observed relationship predicted by the hypothesis in a conceptually consistent way, experimental researchers have also produced process evidence.

Although PEQs are a popular method for obtaining process evidence in the experimental accounting literature, using them has some significant drawbacks that are rarely discussed in the context of controlled experiments. Also, once you carefully consider the experimental method’s strengths, it should become clear that PEQs, at best, offer process evidence that is ancillary to the process evidence that controlled experiments can directly produce.

What are the problems?

PEQs invite socially desirable reporting

Social psychologists, who make great use of questionnaires, have long known about and fought their drawbacks. One of the most pronounced and discussed problems with questionnaires is respondents’ tendency to report socially desirable answers (Nederhof 1985; Podsakoff, MacKenzie, Lee, and Podsakoff 2003; De Jong, Pieters, and Fox 2010; Steenkamp, De Jong, and Baumgartner 2010). In the context of controlled experiments, socially desirable reporting means that once participants arrive at the PEQ, they will say what they think is socially desirable, which is a critical problem if the theory involves a socially sensitive issue (i.e., think about misreporting, honesty, and effort distortion).

Participants may experience fatigue

Accounting researchers often include an array of statements that jointly capture some stable personality trait or mindset among participants. Accordingly, PEQs tend to prolong the duration of controlled experiments substantially. However, participants’ attention span is limited and wears down as the experiment progresses, which is why many do not prolong the experiment more than they have to. When we use PEQs, the chances are relatively high that participants experience fatigue by the time they reach it. Accordingly, they may not fill out the PEQ seriously, leading to more uniform answers (Krosnick, 1999; Galesic and Bosnjak, 2009).

PEQs may implement false memories

Even if participants are still fresh and pay attention by the time they reach the PEQ, they may not perfectly recall what happened or how they thought and felt during the experiment. Ross (1989) and Loftus (1999) provide numerous examples of people reconstructing beliefs under such circumstances using implicit theories rather than relying on their imperfect memory of what actually occurred and what they actually believed at the time. There is also a trail of evidence that false memories and beliefs can arise, sometimes unintentionally, in people’s minds (Gilbert, Krull, and Malone 1990).

Participants may rationalize what happened

PEQs are, by definition, distributed after the experiment has been conducted. Therefore, everything that came before the PEQ influences how participants answer questions and evaluate statements in the PEQ. Suppose participants still have a detailed account in their mind of what happened and are sufficiently motivated to finish the PEQ. In that case, they may fall prey to rationalizing their recent behavior and decisions. In other words, when answering statements in PEQs, participants may resolve any cognitive dissonance by changing their minds about why they behaved the way they did . Suppose we ask participants whether some theoretical explanation and intervention drove their behavior and decisions during the experiment. In that case, it is likely they may agree, even if another explanation is more representative of the truth.

PEQs fail to detect changes in thinking

People often believe that what they are thinking at this moment in time represents how they have thought all along. Therefore, they tend to be unaware of changes in their thinking (Bem and McConnell, 1970). If a participant has changed their opinion during the experiment, they may not report this change in the PEQ. Therefore, using the PEQ to detect changes in thinking may lead to wrong conclusions. For instance, PEQs may reveal no evidence that changes in their thinking drive the changes in participant behavior caused by an intervention while actually, they did.

What about stable personality traits?

Proponents of using PEQs to obtain process evidence may point out that verified instruments measuring stable personality traits are less prone to the abovementioned issues. Precisely, these instruments try to measure participants’ personality type in daily life, which should be independent of what happened during the experiment. If the instrument has been verified and is sufficiently concise and internally consistent, it could provide an excellent way to verify the process underlying an observed relationship.

However, social psychologists have become increasingly concerned about using these instruments specifically and the concept of stable personality traits more generally. As measured by these instruments, many personality traits turn out not to be very stable, in particular, in terms of stability within people over time (Bardi and Zentner 2017; Golsteyn and Schildberg-Hörisch 2017). Some social psychologists have devoted entire careers to proving that the situation is a much more stable driver of people’s behavior than some individual typology (Ross and Nisbett 2011). Thus, even if we use verified personality instruments to showcase process evidence, the relative instability of participants’ responses make it challenging to draw reliable conclusions.

The experiment itself can provide the process evidence

Although questionnaires, outside the context of controlled experiments, give us a cheap way to gain insight into people’s thinking, they rarely provide robust causal evidence of theoretical relationships (Bloomfield, Nelson, and Soltes 2016). It turns out this particular weakness of questionnaires is precisely the comparative advantage of controlled experiments! Controlled experiments introduce interventions and manipulate independent variables with meticulous precision to see how they change participants’ behavior. Depending on how tightly we control the experimental setting, you can find relatively convincing evidence of the process underlying an observed relationship.

Controlled experiments can vary in their ability to provide process evidence because some grant more insight into the process underlying observed relationships than others. One convincing way that controlled experiments can directly obtain process evidence is by featuring interaction effects between different manipulated independent variables. Suppose one manipulation attenuates or strengthens the influence of another manipulation in a conceptually consistent way. In that case, we are surer our theoretical explanation drives the variation observed in the dependent variable(s). Therefore, this type of design we often coin a moderation-of-process design.

Another way that controlled experiments can directly obtain convincing process evidence is by unobtrusively observing participant behavior (Webb, Campbell, Schwartz, and Sechrest 1999). For instance, consider an experiment in which we measure how much time participants spend on a page. The time spent on a particular page could indicate how much attention participants give to a particular piece of information or stimuli. Depending on your hypothesis, it can be used to verify its underlying mechanism. For learning more about this approach to obtaining process evidence, please see this excellent post by Christian Peters.

References

Asay, H. S., R. Guggenmos, K. Kadous, L. Koonce, and R. Libby. 2021. Theory testing and process evidence in accounting experiments. Available at: https://ssrn.com/abstract=3485844.

Bardi, A., and M. Zentner. 2017. Grand challenges for personality and social psychology: Moving beyond the replication crisis. Frontiers in Psychology 8: 2068.

Bloomfield, R. J., M. W. Nelson, and E. F. Soltes. 2016. Gathering data for archival, field, survey and experimental accounting research. Journal of Accounting Research 54 (2): 341-395.

De Jong, M. G., R. Pieters, and J.-P. Fox. 2010. Reducing Social Desirability Bias Through Item Randomized Response: An Application to Measure Underreported Desires. Journal of Marketing Research 47 (1): 14-27.

Gilbert, D. T., D. S. Krull, and P. S. Malone. 1990. Unbelieving the unbelievable: Some problems in the rejection of false information. Journal of Personality and Social Psychology 59 (4): 601-613.

Golsteyn, B., and H. Schildberg-Hörisch. 2017. Challenges in research on preferences and personality traits: Measurement, stability, and inference. Journal of Economic Psychology 60: 1-6.

Loftus, E. F. 1999. Suggestion, Imagination, and The Transformation of Reality, edited by A. A. Stone, C. A. Bachrach, J. B. Jobe, H. S. Kurtzman and V. S. Cain: Psychology Press, 201-210.

Nederhof, A. J. 1985. Methods of coping with social desirability bias: A review. European Journal of Social Psychology 15 (3): 263-280.

Podsakoff, P. M., S. B. MacKenzie, J.-Y. Lee, and N. P. Podsakoff. 2003. Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology 88 (5): 879.

Ross, L., and R. E. Nisbett. 2011. The Person and the Situation: Perspectives of Social Psychology: Pinter & Martin Publishers.

Ross, M. 1989. Relation of implicit theories to the construction of personal histories. Psychological Review 96 (2): 341-357.

Steenkamp, J.-B. E., M. G. De Jong, and H. Baumgartner. 2010. Socially desirable response tendencies in survey research. Journal of Marketing Research 47 (2): 199-214.

Webb, E. J., D. T. Campbell, R. D. Schwartz, and L. Sechrest. 1999. Unobtrusive Measures. Vol. 2: Sage Publications.