# Interaction (statistics)

Last updated

In statistics, an interaction may arise when considering the relationship among three or more variables, and describes a situation in which the effect of one causal variable on an outcome depends on the state of a second causal variable (that is, when effects of the two causes are not additive). [1] [2] Although commonly thought of in terms of causal relationships, the concept of an interaction can also describe non-causal associations. Interactions are often considered in the context of regression analyses or factorial experiments.

## Contents

The presence of interactions can have important implications for the interpretation of statistical models. If two variables of interest interact, the relationship between each of the interacting variables and a third "dependent variable" depends on the value of the other interacting variable. In practice, this makes it more difficult to predict the consequences of changing the value of a variable, particularly if the variables it interacts with are hard to measure or difficult to control.

The notion of "interaction" is closely related to that of moderation that is common in social and health science research: the interaction between an explanatory variable and an environmental variable suggests that the effect of the explanatory variable has been moderated or modified by the environmental variable. [1]

## Introduction

An interaction variable or interaction feature is a variable constructed from an original set of variables to try to represent either all of the interaction present or some part of it. In exploratory statistical analyses it is common to use products of original variables as the basis of testing whether interaction is present with the possibility of substituting other more realistic interaction variables at a later stage. When there are more than two explanatory variables, several interaction variables are constructed, with pairwise-products representing pairwise-interactions and higher order products representing higher order interactions.

Thus, for a response Y and two variables x1 and x2 an additive model would be:

${\displaystyle Y=c+ax_{1}+bx_{2}+{\text{error}}\,}$

In contrast to this,

${\displaystyle Y=c+ax_{1}+bx_{2}+d(x_{1}\times x_{2})+{\text{error}}\,}$

is an example of a model with an interaction between variables x1 and x2 ("error" refers to the random variable whose value is that by which Y differs from the expected value of Y; see errors and residuals in statistics). Often, models are presented without the interaction term ${\displaystyle d(x_{1}\times x_{2})}$, but this confounds the main effect and interaction effect (i.e., without specifying the interaction term, it is possible that any main effect found is actually due to an interaction).

## In modeling

### In ANOVA

A simple setting in which interactions can arise is a two-factor experiment analyzed using Analysis of Variance (ANOVA). Suppose we have two binary factors A and B. For example, these factors might indicate whether either of two treatments were administered to a patient, with the treatments applied either singly, or in combination. We can then consider the average treatment response (e.g. the symptom levels following treatment) for each patient, as a function of the treatment combination that was administered. The following table shows one possible situation:

B = 0B = 1
A = 067
A = 145

In this example, there is no interaction between the two treatments their effects are additive. The reason for this is that the difference in mean response between those subjects receiving treatment A and those not receiving treatment A is 2 regardless of whether treatment B is administered (2 = 4  6) or not (2 = 5  7). Note that it automatically follows that the difference in mean response between those subjects receiving treatment B and those not receiving treatment B is the same regardless of whether treatment A is administered (7  6 = 5  4).

In contrast, if the following average responses are observed

B = 0B = 1
A = 014
A = 176

then there is an interaction between the treatments their effects are not additive. Supposing that greater numbers correspond to a better response, in this situation treatment B is helpful on average if the subject is not also receiving treatment A, but is detrimental on average if given in combination with treatment A. Treatment A is helpful on average regardless of whether treatment B is also administered, but it is more helpful in both absolute and relative terms if given alone, rather than in combination with treatment B. Similar observations are made for this particular example in the next section.

### Qualitative and quantitative interactions

In many applications it is useful to distinguish between qualitative and quantitative interactions. [3] A quantitative interaction between A and B is a situation where the magnitude of the effect of B depends on the value of A, but the direction of the effect of B is constant for all A. A qualitative interaction between A and B refers to a situation where both the magnitude and direction of each variable's effect can depend on the value of the other variable.

The table of means on the left, below, shows a quantitative interaction treatment A is beneficial both when B is given, and when B is not given, but the benefit is greater when B is not given (i.e. when A is given alone). The table of means on the right shows a qualitative interaction. A is harmful when B is given, but it is beneficial when B is not given. Note that the same interpretation would hold if we consider the benefit of B based on whether A is given.

B = 0B = 1B = 0B = 1
A = 021A = 026
A = 153A = 153

The distinction between qualitative and quantitative interactions depends on the order in which the variables are considered (in contrast, the property of additivity is invariant to the order of the variables). In the following table, if we focus on the effect of treatment A, there is a quantitative interaction giving treatment A will improve the outcome on average regardless of whether treatment B is or is not already being given (although the benefit is greater if treatment A is given alone). However, if we focus on the effect of treatment B, there is a qualitative interaction giving treatment B to a subject who is already receiving treatment A will (on average) make things worse, whereas giving treatment B to a subject who is not receiving treatment A will improve the outcome on average.

B = 0B = 1
A = 014
A = 176

In its simplest form, the assumption of treatment unit additivity states that the observed response yij from experimental unit i when receiving treatment j can be written as the sum yij = yi + tj. [4] [5] [6] The assumption of unit treatment additivity implies that every treatment has exactly the same additive effect on each experimental unit. Since any given experimental unit can only undergo one of the treatments, the assumption of unit treatment additivity is a hypothesis that is not directly falsifiable, according to Cox[ citation needed ] and Kempthorne.[ citation needed ]

However, many consequences of treatment-unit additivity can be falsified.[ citation needed ] For a randomized experiment, the assumption of treatment additivity implies that the variance is constant for all treatments. Therefore, by contraposition, a necessary condition for unit treatment additivity is that the variance is constant.[ citation needed ]

The property of unit treatment additivity is not invariant under a change of scale,[ citation needed ] so statisticians often use transformations to achieve unit treatment additivity. If the response variable is expected to follow a parametric family of probability distributions, then the statistician may specify (in the protocol for the experiment or observational study) that the responses be transformed to stabilize the variance. [7] In many cases, a statistician may specify that logarithmic transforms be applied to the responses, which are believed to follow a multiplicative model. [5] [8]

The assumption of unit treatment additivity was enunciated in experimental design by Kempthorne[ citation needed ] and Cox[ citation needed ]. Kempthorne's use of unit treatment additivity and randomization is similar to the design-based analysis of finite population survey sampling.

In recent years, it has become common[ citation needed ] to use the terminology of Donald Rubin, which uses counterfactuals. Suppose we are comparing two groups of people with respect to some attribute y. For example, the first group might consist of people who are given a standard treatment for a medical condition, with the second group consisting of people who receive a new treatment with unknown effect. Taking a "counterfactual" perspective, we can consider an individual whose attribute has value y if that individual belongs to the first group, and whose attribute has value τ(y) if the individual belongs to the second group. The assumption of "unit treatment additivity" is that τ(y) = τ, that is, the "treatment effect" does not depend on y. Since we cannot observe both y and τ(y) for a given individual, this is not testable at the individual level. However, unit treatment additivity implies that the cumulative distribution functions F1 and F2 for the two groups satisfy F2(y)  = F1(y  τ), as long as the assignment of individuals to groups 1 and 2 is independent of all other factors influencing y (i.e. there are no confounders). Lack of unit treatment additivity can be viewed as a form of interaction between the treatment assignment (e.g. to groups 1 or 2), and the baseline, or untreated value of y.

### Categorical variables

Sometimes the interacting variables are categorical variables rather than real numbers and the study might then be dealt with as an analysis of variance problem. For example, members of a population may be classified by religion and by occupation. If one wishes to predict a person's height based only on the person's religion and occupation, a simple additive model, i.e., a model without interaction, would add to an overall average height an adjustment for a particular religion and another for a particular occupation. A model with interaction, unlike an additive model, could add a further adjustment for the "interaction" between that religion and that occupation. This example may cause one to suspect that the word interaction is something of a misnomer.

Statistically, the presence of an interaction between categorical variables is generally tested using a form of analysis of variance (ANOVA). If one or more of the variables is continuous in nature, however, it would typically be tested using moderated multiple regression. [9] This is so-called because a moderator is a variable that affects the strength of a relationship between two other variables.

### Designed experiments

Genichi Taguchi contended [10] that interactions could be eliminated from a system by appropriate choice of response variable and transformation. However George Box and others have argued that this is not the case in general. [11]

### Model size

Given n predictors, the number of terms in a linear model that includes a constant, every predictor, and every possible interaction is ${\displaystyle {\tbinom {n}{0}}+{\tbinom {n}{1}}+{\tbinom {n}{2}}+\cdots +{\tbinom {n}{n}}=2^{n}}$. Since this quantity grows exponentially, it readily becomes impractically large. One method to limit the size of the model is to limit the order of interactions. For example, if only two-way interactions are allowed, the number of terms becomes ${\displaystyle {\tbinom {n}{0}}+{\tbinom {n}{1}}+{\tbinom {n}{2}}=1+{\tfrac {1}{2}}n+{\tfrac {1}{2}}n^{2}}$. The below table shows the number of terms for each number of predictors and maximum order of interaction.

Number of terms
PredictorsIncluding up to m-way interactions
2345
122222
244444
378888
41115161616
51626313232
62242576364
7296499120128
83793163219256
946130256382512
10561763866381,024
11672325621,0242,048
12792997941,5864,096
13923781,0932,3808,192
141064701,4713,47316,384
151215761,9414,94432,768
202111,3516,19621,7001,048,576
253262,62615,27668,40633,554,432
501,27620,876251,1762,369,9361015
1005,051166,7514,087,97679,375,4961030
1,000500,501166,667,5011010101210300

### In regression

The most general approach to modeling interaction effects involves regression, starting from the elementary version given above:

${\displaystyle Y=c+ax_{1}+bx_{2}+d(x_{1}\times x_{2})+{\text{error}}\,}$

where the interaction term ${\displaystyle (x_{1}\times x_{2})}$ could be formed explicitly by multiplying two (or more) variables, or implicitly using factorial notation in modern statistical packages such as Stata. The components x1 and x2 might be measurements or {0,1} dummy variables in any combination. Interactions involving a dummy variable multiplied by a measurement variable are termed slope dummy variables, [12] because they estimate and test the difference in slopes between groups 0 and 1.

When measurement variables are employed in interactions, it is often desirable to work with centered versions, where the variable's mean (or some other reasonably central value) is set as zero. Centering can make the main effects in interaction models more interpretable, as it reduces the multicollinearity between the interaction term and the main effects. [13] The coefficient a in the equation above, for example, represents the effect of x1 when x2 equals zero.

Regression approaches to interaction modeling are very general because they can accommodate additional predictors, and many alternative specifications or estimation strategies beyond ordinary least squares. Robust, quantile, and mixed-effects (multilevel) models are among the possibilities, as is generalized linear modeling encompassing a wide range of categorical, ordered, counted or otherwise limited dependent variables. The graph depicts an education*politics interaction, from a probability-weighted logit regression analysis of survey data. [14]

## Interaction plots

Interaction plots show possible interactions among variables.

### Example: Interaction of species and air temperature and their effect on body temperature

Consider a study of the body temperature of different species at different air temperatures, in degrees Fahrenheit. The data are shown in the table below.

The interaction plot may use either the air temperature or the species as the x axis. The second factor is represented by lines on the interaction plot.

There is an interaction between the two factors (air temperature and species) in their effect on the response (body temperature), because the effect of the air temperature depends on the species. The interaction is indicated on the plot because the lines are not parallel.

### Example: effect of stroke severity and treatment on recovery

As a second example, consider a clinical trial on the interaction between stroke severity and the efficacy of a drug on patient survival. The data are shown in the table below.

In the interaction plot, the lines for the mild and moderate stroke groups are parallel, indicating that the drug has the same effect in both groups, so there is no interaction. The line for the severe stroke group is not parallel to the other lines, indicating that there is an interaction between stroke severity and drug effect on survival. The line for the severe stroke group is flat, indicating that, among these patients, there is no difference in survival between the drug and placebo treatments. In contrast, the lines for the mild and moderate stroke groups slope down to the right, indicating that, among these patients, the placebo group has lower survival than drug-treated group.

## Hypothesis tests for interactions

Analysis of variance and regression analysis are used to test for significant interactions.

Is the yield of good cookies affected by the baking temperature and time in the oven? The table shows data for 8 batches of cookies.

The data show that the yield of good cookies is best when either (i) temperature is high and time in the oven is short, or (ii) temperature is low and time in the oven is long. If the cookies are left in the oven for a long time at a high temperature, there are burnt cookies and the yield is low.

From the graph and the data, it is clear that the lines are not parallel, indicating that there is an interaction. This can be tested using analysis of variance (ANOVA). The first ANOVA model will not include the interaction term. That is, the first ANOVA model ignores possible interaction. The second ANOVA model will include the interaction term. That is, the second ANOVA model explicitly performs a hypothesis test for interaction.

#### ANOVA model 1: no interaction term; yield ~ temperature + time

In the ANOVA model that ignores interaction, neither temperature nor time has a significant effect on yield (p=0.91), which is clearly the incorrect conclusion. The more appropriate ANOVA model should test for possible interaction.

#### ANOVA model 2: include interaction term; yield ~ temperature * time

The temperature:time interaction term is significant (p=0.000180). Based on the interaction test and the interaction plot, it appears that the effect of time on yield depends on temperature and vice versa.

## Examples

Real-world examples of interaction include:

• Interaction between adding sugar to coffee and stirring the coffee. Neither of the two individual variables has much effect on sweetness but a combination of the two does.
• Interaction between adding carbon to steel and quenching. Neither of the two individually has much effect on strength but a combination of the two has a dramatic effect.
• Interaction between smoking and inhaling asbestos fibres: Both raise lung carcinoma risk, but exposure to asbestos multiplies the cancer risk in smokers and non-smokers. Here, the joint effect of inhaling asbestos and smoking is higher than the sum of both effects. [15]
• Interaction between genetic risk factors for type 2 diabetes and diet (specifically, a "western" dietary pattern). The western dietary pattern was shown to increase diabetes risk for subjects with a high "genetic risk score", but not for other subjects. [16]
• Interaction between education and political orientation, affecting general-public perceptions about climate change. For example, US surveys often find that acceptance of the reality of anthropogenic climate change rises with education among moderate or liberal survey respondents, but declines with education among the most conservative. [17] [18] Similar interactions have been observed to affect some non-climate science or environmental perceptions, [19] and to operate with science literacy or other knowledge indicators in place of education. [20] [21]

## Related Research Articles

Analysis of variance (ANOVA) is a collection of statistical models and their associated estimation procedures used to analyse the differences among means. ANOVA was developed by the statistician Ronald Fisher. ANOVA is based on the law of total variance, where the observed variance in a particular variable is partitioned into components attributable to different sources of variation. In its simplest form, ANOVA provides a statistical test of whether two or more population means are equal, and therefore generalizes the t-test beyond two means.

Heritability is a statistic used in the fields of breeding and genetics that estimates the degree of variation in a phenotypic trait in a population that is due to genetic variation between individuals in that population. It measures how much of the variation of a trait can be attributed to variation of genetic factors, as opposed to variation of environmental factors. The concept of heritability can be expressed in the form of the following question: "What is the proportion of the variation in a given trait within a population that is not explained by the environment or random chance?"

In statistics and econometrics, particularly in regression analysis, a dummy variable is one that takes only the value 0 or 1 to indicate the absence or presence of some categorical effect that may be expected to shift the outcome. They can be thought of as numeric stand-ins for qualitative facts in a regression model, sorting data into mutually exclusive categories.

An F-test is any statistical test in which the test statistic has an F-distribution under the null hypothesis. It is most often used when comparing statistical models that have been fitted to a data set, in order to identify the model that best fits the population from which the data were sampled. Exact "F-tests" mainly arise when the models have been fitted to the data using least squares. The name was coined by George W. Snedecor, in honour of Sir Ronald A. Fisher. Fisher initially developed the statistic as the variance ratio in the 1920s.

Analysis of covariance (ANCOVA) is a general linear model which blends ANOVA and regression. ANCOVA evaluates whether the means of a dependent variable (DV) are equal across levels of a categorical independent variable (IV) often called a treatment, while statistically controlling for the effects of other continuous variables that are not of primary interest, known as covariates (CV) or nuisance variables. Mathematically, ANCOVA decomposes the variance in the DV into variance explained by the CV(s), variance explained by the categorical IV, and residual variance. Intuitively, ANCOVA can be thought of as 'adjusting' the DV by the group means of the CV(s).

Linear trend estimation is a statistical technique to aid interpretation of data. When a series of measurements of a process are treated as, for example, a sequences or time series, trend estimation can be used to make and justify statements about tendencies in the data, by relating the measurements to the times at which they occurred. This model can then be used to describe the behaviour of the observed data, without explaining it. In this case linear trend estimation expresses data as a linear function of time, and can also be used to determine the significance of differences in a set of data linked by a categorical factor. An example of the latter from biomedical science would be levels of a molecule in the blood or tissues of patients with incrementally worsening disease – such as mild, moderate and severe. This is in contrast to an ANOVA, which is reserved for three or more independent groups.

In the statistical theory of the design of experiments, blocking is the arranging of experimental units in groups (blocks) that are similar to one another. Blocking can be used to tackle the problem of pseudoreplication.

In statistics, a full factorial experiment is an experiment whose design consists of two or more factors, each with discrete possible values or "levels", and whose experimental units take on all possible combinations of these levels across all such factors. A full factorial design may also be called a fully crossed design. Such an experiment allows the investigator to study the effect of each factor on the response variable, as well as the effects of interactions between factors on the response variable.

Multilevel models are statistical models of parameters that vary at more than one level. An example could be a model of student performance that contains measures for individual students as well as measures for classrooms within which the students are grouped. These models can be seen as generalizations of linear models, although they can also extend to non-linear models. These models became much more popular after sufficient computing power and software became available.

In the design of experiments and analysis of variance, a main effect is the effect of an independent variable on a dependent variable averaged across the levels of any other independent variables. The term is frequently used in the context of factorial designs and regression models to distinguish main effects from interaction effects.

Analysis of variance – simultaneous component analysis is a method that partitions variation and enables interpretation of these partitions by SCA, a method that is similar to principal components analysis (PCA). This method is a multivariate or even megavariate extension of analysis of variance (ANOVA). The variation partitioning is similar to ANOVA. Each partition matches all variation induced by an effect or factor, usually a treatment regime or experimental condition. The calculated effect partitions are called effect estimates. Because even the effect estimates are multivariate, interpretation of these effects estimates is not intuitive. By applying SCA on the effect estimates one gets a simple interpretable result. In case of more than one effect this method estimates the effects in such a way that the different effects are not correlated.

In statistics, one-way analysis of variance is a technique that can be used to compare whether two samples means are significantly different or not. This technique can be used only for numerical response data, the "Y", usually one variable, and numerical or (usually) categorical input data, the "X", always one variable, hence "one-way".

Repeated measures design is a research design that involves multiple measures of the same variable taken on the same or matched subjects either under different conditions or over two or more time periods. For instance, repeated measurements are collected in a longitudinal study in which change over time is assessed.

The following is a glossary of terms. It is not intended to be all-inclusive.

In statistics, a mixed-design analysis of variance model, also known as a split-plot ANOVA, is used to test for differences between two or more independent groups whilst subjecting participants to repeated measures. Thus, in a mixed-design ANOVA model, one factor is a between-subjects variable and the other is a within-subjects variable. Thus, overall, the model is a type of mixed-effects model.

In statistics and regression analysis, moderation occurs when the relationship between two variables depends on a third variable. The third variable is referred to as the moderator variable or simply the moderator. The effect of a moderating variable is characterized statistically as an interaction; that is, a categorical or quantitative variable that affects the direction and/or strength of the relation between dependent and independent variables. Specifically within a correlational analysis framework, a moderator is a third variable that affects the zero-order correlation between two other variables, or the value of the slope of the dependent variable on the independent variable. In analysis of variance (ANOVA) terms, a basic moderator effect can be represented as an interaction between a focal independent variable and a factor that specifies the appropriate conditions for its operation.

In statistics, Tukey's test of additivity, named for John Tukey, is an approach used in two-way ANOVA to assess whether the factor variables are additively related to the expected value of the response variable. It can be applied when there are no replicated values in the data set, a situation in which it is impossible to directly estimate a fully general non-additive regression structure and still have information left to estimate the error variance. The test statistic proposed by Tukey has one degree of freedom under the null hypothesis, hence this is often called "Tukey's one-degree-of-freedom test."

In randomized statistical experiments, generalized randomized block designs (GRBDs) are used to study the interaction between blocks and treatments. For a GRBD, each treatment is replicated at least two times in each block; this replication allows the estimation and testing of an interaction term in the linear model.

In statistics, the principle of marginality is the fact that the average effects, of variables in an analysis are marginal to their interaction effect—that is, the main effect of one explanatory variable captures the effect of that variable averaged over all values of a second explanatory variable whose value influences the first variable's effect. The principle of marginality implies that, in general, it is wrong to test, estimate, or interpret main effects of explanatory variables where the variables interact or, similarly, to model interaction effects but delete main effects that are marginal to them. While such models are interpretable, they lack applicability, as they ignore the dependence of a variable's effect upon another variable's value.

In statistics, the two-way analysis of variance (ANOVA) is an extension of the one-way ANOVA that examines the influence of two different categorical independent variables on one continuous dependent variable. The two-way ANOVA not only aims at assessing the main effect of each independent variable but also if there is any interaction between them.

## References

1. Dodge, Y. (2003). . Oxford University Press. ISBN   978-0-19-920613-1.
2. Cox, D.R. (1984). "Interaction". International Statistical Review. 52 (1): 1–25. doi:10.2307/1403235. JSTOR   1403235.
3. Peto, D. P. (1982). "Statistical aspects of cancer trials". Treatment of Cancer (First ed.). London: Chapman and Hall. ISBN   0-412-21850-X.
4. Kempthorne, Oscar (1979). The Design and Analysis of Experiments (Corrected reprint of (1952) Wiley ed.). Robert E. Krieger. ISBN   978-0-88275-105-4.
5. Cox, David R. (1958). Planning of experiments. Chapter 2. ISBN   0-471-57429-5.
6. Hinkelmann, Klaus and Kempthorne, Oscar (2008). Design and Analysis of Experiments, Volume I: Introduction to Experimental Design (Second ed.). Wiley. Chapters 5-6. ISBN   978-0-471-72756-9.CS1 maint: multiple names: authors list (link)
7. Hinkelmann, Klaus and Kempthorne, Oscar (2008). Design and Analysis of Experiments, Volume I: Introduction to Experimental Design (Second ed.). Wiley. Chapters 7-8. ISBN   978-0-471-72756-9.CS1 maint: multiple names: authors list (link)
8. Bailey, R. A. (2008). Design of Comparative Experiments. Cambridge University Press. ISBN   978-0-521-68357-9. Pre-publication chapters are available on-line.
9. Overton, R. C. (2001). "Moderated multiple regression for interactions involving categorical variables: a statistical control for heterogeneous variance across two groups". Psychol Methods. 6 (3): 218–33. doi:10.1037/1082-989X.6.3.218. PMID   11570229.
10. "Design of Experiments - Taguchi Experiments". www.qualitytrainingportal.com. Retrieved 2015-11-27.
11. George E. P. Box (1990). "Do interactions matter?" (PDF). Quality Engineering. 2: 365–369. doi:10.1080/08982119008962728. Archived from the original (PDF) on 2010-06-10. Retrieved 2009-07-28.
12. Hamilton, L.C. 1992. Regression with Graphics: A Second Course in Applied Statistics. Pacific Grove, CA: Brooks/Cole. ISBN   978-0534159009
13. Iacobucci, Dawn; Schneider, Matthew J.; Popovich, Deidre L.; Bakamitsos, Georgios A. (2016). "Mean centering helps alleviate "micro" but not "macro" multicollinearity". Behavior Research Methods. 48 (4): 1308–1317. doi:. ISSN   1554-3528.
14. Hamilton, L.C.; Saito, K. (2015). "A four-party view of U.S. environmental concern". Environmental Politics. 24 (2): 212–227. doi:10.1080/09644016.2014.976485. S2CID   154762226.
15. Lee, P. N. (2001). "Relation between exposure to asbestos and smoking jointly and the risk of lung cancer". Occupational and Environmental Medicine. 58 (3): 145–53. doi:10.1136/oem.58.3.145. PMC  . PMID   11171926.
16. Lu, Q.; et al. (2009). "Genetic predisposition, Western dietary pattern, and the risk of type 2 diabetes in men". Am J Clin Nutr. 89 (5): 1453–1458. doi:10.3945/ajcn.2008.27249. PMC  . PMID   19279076.
17. Hamilton, L.C. (2011). "Education, politics and opinions about climate change: Evidence for interaction effects". Climatic Change . 104 (2): 231–242. doi:10.1007/s10584-010-9957-8. S2CID   16481640.
18. McCright, A. M. (2011). "Political orientation moderates Americans' beliefs and concern about climate change". Climatic Change . 104 (2): 243–253. doi:10.1007/s10584-010-9946-y. S2CID   152795205.
19. Hamilton, Lawrence C.; Saito, Kei (2015). "A four-party view of US environmental concern". Environmental Politics. 24 (2): 212–227. doi:10.1080/09644016.2014.976485. S2CID   154762226.
20. Kahan, D.M.; Jenkins-Smith, H.; Braman, D. (2011). "Cultural cognition of scientific consensus". Journal of Risk Research. 14 (2): 147–174. doi:10.1080/13669877.2010.511246. hdl:. S2CID   216092368.
21. Hamilton, L.C.; Cutler, M.J.; Schaefer, A. (2012). "Public knowledge and concern about polar-region warming". Polar Geography . 35 (2): 155–168. doi:10.1080/1088937X.2012.684155. S2CID   12437794.