Why ANOVA and linear regression are the same
If your doctoral training was anything like ours, you have learned to conduct ANOVAs for one class of experiments (i.e., behavioral experiments), while running linear regressions for another class of experiments (i.e., economic experiments). However, there was not a single class that put it all together. When asked, people may advocate the use of one analysis over the other, but give vague reasons when pressed. It was not until we started comparing ANOVA and linear regression ourselves that we realized how closely-related they are.
In this post, we use a simple example to show you the equivalence between ANOVA and a linear regression. Our example features a two-way interaction effect between two binary categorical variables. Therefore, it should be representative for the vast majority of accounting experiments.
When comparing ANOVA and linear regression, it is essential to distinguish two types of of coding: Effect coding and dummy coding. The first type of coding contrasts group means with the grand mean, whereas the last type of coding contrasts group means with a specified reference group. If you have multiple categorical variables in your model, often the choice between effect coding and dummy coding for your interpretation does not matter. However, when you have an interaction with two categorical variables, the main effects either represent “true” main effects (for effect coding) or simple effects (for dummy coding).
We use publicly available data from the Stata 16 manual (i.e., fitness.dta). We do not aim to use this dataset to test some hypothesis. Instead, our goal is to estimate a basic empirical model predicting the continuous dependent variable “hours,” which are the number of hours an individual exercises per day, using two binary categorical variables (i.e., “smoke” and “single”) and their interaction.
We first load the data in Stata 16 and tabulate the means to get an impression of how “hours” varies across the two categorical variables “smoke” and “single.”
. webuse fitness, clear
. table smoke single, c(mean hours) row col
-----------------------------------------
dummy for | dummy for not married
smoking | married single Total
-----------+-----------------------------
nonsmoking | .6644013 1.075681 .9116104
smoking | .4489347 .6688333 .5891583
Total | .6450533 1.033642 .8800172
-----------------------------------------
The output displays the means for each category, sub-category, and even sub-sub-category. Also, the output reports a grand mean of 0.8800172.
We run a two-way ANOVA immediately followed by its underlying linear regression model. The ANOVA automatically uses effect coding for the categorical variables “smoke” and “single.” This means that the categories are coded with 1’s and -1’s so that each category’s mean is compared to the grand mean of the sample. In the linear regression, the categorical variables are dummy coded, which means that each category’s mean is compared to the reference group’s mean. In this case, the reference group is the group that does not smoke and that are not single.
. anova hours smoke##single
Number of obs = 19,831 R-squared = 0.0424
Root MSE = 1.02876 Adj R-squared = 0.0423
Source | Partial SS df MS F Prob>F
-------------+----------------------------------------------------
Model | 929.4697 3 309.82323 292.74 0.0000
|
smoke | 157.38411 1 157.38411 148.71 0.0000
single | 161.89951 1 161.89951 152.97 0.0000
smoke#single | 14.884657 1 14.884657 14.06 0.0002
|
Residual | 20983.97 19,827 1.0583533
-------------+----------------------------------------------------
Total | 21913.44 19,830 1.105065
. regress
Source | SS df MS Number of obs = 19,831
-------------+---------------------------------- F(3, 19827) = 292.74
Model | 929.4697 3 309.823233 Prob > F = 0.0000
Residual | 20983.9702 19,827 1.05835327 R-squared = 0.0424
-------------+---------------------------------- Adj R-squared = 0.0423
Total | 21913.4399 19,830 1.10506505 Root MSE = 1.0288
---------------------------------------------------------------------------------
hours | Coef. Std. Err. t P>|t| [95% Conf. Interval]
----------------+----------------------------------------------------------------
smoke | -.2154666 .0406406 -5.30 0.000 -.2951254 -.1358077
|
single | .4112795 .0157081 26.18 0.000 .3804902 .4420687
|
smoke#single | -.1913809 .0510322 -3.75 0.000 -.2914083 -.0913535
|
_cons | .6644013 .0121783 54.56 0.000 .6405307 .6882719
---------------------------------------------------------------------------------
. test 1.smoke#1.single
( 1) 1.smoke#1.single = 0
F( 1, 19827) = 14.06
Prob > F = 0.0002
Both the ANOVA and linear regression output report an identical F-statistic for the empirical model (F(3,19827) = 292.74, two-tailed p-value < 0.001). This should already be a sign that the analyses are essentially equivalent. Specifically, they present the same information but in different ways.
We can use the linear regression output to retrieve each of the means by departing from the mean of the reference group, which is the constant of 0.6644013, and adding regression coefficients. For instance, we can retrieve the mean for the category of people who smoke but are not single by adding the regression coefficient of “smoke” to the regression's constant (0.6644013-0.2154666 = 0.4489347).
It is not very complicated to also make the statistical tests reported in the linear regression equivalent to the tests reported in the ANOVA (or at least show that they are the same). To achieve this, we will estimate the linear regression not with dummy variables but with variables that are effect coded (just like the ANOVA). We include the extension “_rec” after these new variables to highlight that they are effect coded.
. regress hours smoke_rec single_rec smoke_single_rec
Source | SS df MS Number of obs = 19,831
-------------+---------------------------------- F(3, 19827) = 292.74
Model | 929.4697 3 309.823233 Prob > F = 0.0000
Residual | 20983.9702 19,827 1.05835327 R-squared = 0.0424
-------------+---------------------------------- Adj R-squared = 0.0423
Total | 21913.4399 19,830 1.10506505 Root MSE = 1.0288
----------------------------------------------------------------------------------
hours | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-----------------+----------------------------------------------------------------
smoke_rec | -.1555785 .0127581 -12.19 0.000 -.1805854 -.1305716
single_rec | .1577945 .0127581 12.37 0.000 .1327877 .1828014
smoke_single_rec | -.0478452 .0127581 -3.75 0.000 -.0728521 -.0228384
_cons | .7144625 .0127581 56.00 0.000 .6894557 .7394694
----------------------------------------------------------------------------------
.
. test smoke_rec
( 1) smoke_rec = 0
F( 1, 19827) = 148.71
Prob > F = 0.0000
. test single_rec
( 1) single_rec = 0
F( 1, 19827) = 152.97
Prob > F = 0.0000
. test smoke_single_rec
( 1) smoke_single_rec = 0
F( 1, 19827) = 14.06
Prob > F = 0.0002
When we run the test command for each coefficient, which just tests for linear effects after the estimation, we obtain the same F-statistics reported in the original ANOVA.
Our example suggests it may be commendable not to cling too strongly to one's preference for either linear regression or ANOVA. At least for relatively simple, two-way interaction models, which are popular in experimental accounting, the two approaches are econometrically the same. However, for other, often more complicated, analyses, things may be different and some experts suggest using one type of analysis over another (e.g., Gelman, 2005). Thus, we suggest you try this little exercise yourself with your experimental data.
References
Gelman, A. (2005). Analysis of variance—why it is more important than ever. Annals of statistics, 33(1), 1-53.
How to reference this online article?
Peters, C., & Van Pelt, V. F. J. (2021, February 26). Why anova and linear regression are the same. Accounting Experiments, Available at: https://www.accountingexperiments.com/post/anova-regression/.
