Spillover (experiment)

Last updated October 26, 2023

In experiments, a spillover is an indirect effect on a subject not directly treated by the experiment. These effects are useful for policy analysis but complicate the statistical analysis of experiments.

Analysis of spillover effects involves relaxing the non-interference assumption, or SUTVA (Stable Unit Treatment Value Assumption). This assumption requires that subject i's revelation of its potential outcomes depends only on that subject i's own treatment status, and is unaffected by another subject j's treatment status. In ordinary settings where the researcher seeks to estimate the average treatment effect ( ${\widehat {ATE}}$ ), violation of the non-interference assumption means that traditional estimators for the ATE, such as difference-in-means, may be biased. However, there are many real-world instances where a unit's revelation of potential outcomes depend on another unit's treatment assignment, and analyzing these effects may be just as important as analyzing the direct effect of treatment.

One solution to this problem is to redefine the causal estimand of interest by redefining a subject's potential outcomes in terms of one's own treatment status and related subjects' treatment status. The researcher can then analyze various estimands of interest separately. One important assumption here is that this process captures all patterns of spillovers, and that there are no unmodeled spillovers remaining (ex. spillovers occur within a two-person household but not beyond).

Once the potential outcomes are redefined, the rest of the statistical analysis involves modeling the probabilities of being exposed to treatment given some schedule of treatment assignment, and using inverse probability weighting (IPW) to produce unbiased (or asymptotically unbiased) estimates of the estimand of interest.

Examples of spillover effects

Spillover effects can occur in a variety of different ways. Common applications include the analysis of social network spillovers and geographic spillovers. Examples include the following:

Communication: An intervention that conveys information about a technology or product can influence the take-up decisions of others in their network if it diffuses beyond the initial user.^[1]
Competition: Job placement assistance for young job seekers may influence the job market prospects of individuals who did not receive the training but are competing for the same jobs.^[2]
Contagion: Receiving deworming drugs can decrease other's likelihood of contracting the disease.^[3]
Deterrence: Information about government audits in specific municipalities can spread to nearby municipalities.^[4]
Displacement: A hotspot policing intervention that increases policing presence on a given street can lead to the displacement of crime onto nearby untreated streets.^[5]
Reallocation of resources: A hotspot policing intervention that increases policing presence on a given street can decrease police presence on nearby streets.
Social comparison: A program that randomizes individuals to receive a voucher to move to a new neighborhood can additionally influence the control group's beliefs about their housing conditions.^[6]

In such examples, treatment in a randomized-control trial can have a direct effect on those who receive the intervention and also a spillover effect on those who were not directly treated.

Statistical issues

Estimating spillover effects in experiments introduces three statistical issues that researchers must take into account.

Relaxing the non-interference assumption

One key assumption for unbiased inference is the non-interference assumption, which posits that an individual's potential outcomes are only revealed by their own treatment assignment and not the treatment assignment of others.^[7] This assumption has also been called the Individualistic Treatment Response^[8] or the stable unit treatment value assumption. Non-interference is violated when subjects can communicate with each other about their treatments, decisions, or experiences, thereby influencing each other's potential outcomes. If the non-interference assumption does not hold, units no longer have just two potential outcomes (treated and control), but a variety of other potential outcomes that depend on other units’ treatment assignments,^[9] which complicates the estimation of the average treatment effect.

Estimating spillover effects requires relaxing the non-interference assumption. This is because a unit's outcomes depend not only on its treatment assignment but also on the treatment assignment of its neighbors. The researcher must posit a set of potential outcomes that limit the type of interference. As an example, consider an experiment that sends out political information to undergraduate students to increase their political participation. If the study population consists of all students living with a roommate in a college dormitory, one can imagine four sets of potential outcomes, depending on whether the student or their partner received the information (assume no spillover outside of each two-person room):

Y_0,0 refers to an individual's potential outcomes when they are not treated (0) and neither was their roommate (0).
Y_0,1 refers to an individual's potential outcome when they are not treated (0) but their roommate was treated (1).
Y_1,0 refers to an individual's potential outcome when they are treated (1) but their roommate was not treated (0).
Y_1,1 refers to an individual's potential outcome when they are treated (1) and their roommate was treated (1).

Now an individual's outcomes are influenced by both whether they received the treatment and whether their roommate received the treatment. We can estimate one type of spillover effect by looking at how one's outcomes change depending on whether their roommate received the treatment or not, given the individual did not receive treatment directly. This would be captured by the difference Y_0,1- Y_0,0. Similarly, we can measure how ones’ outcomes change depending on their roommate's treatment status, when the individual themselves are treated. This amounts to taking the difference Y_1,1- Y_1,0.

While researchers typically embrace experiments because they require less demanding assumptions, spillovers can be “unlimited in extent and impossible to specify in form.” ^[10] The researcher must make specific assumptions about which types of spillovers are operative. One can relax the non-interference assumption in various ways depending on how spillovers are thought to occur in a given setting. One way to model spillover effects is a binary indicator for whether an immediate neighbor was also treated, as in the example above. One can also posit spillover effects that depend on the number of immediate neighbors that were also treated, also known as k-level effects.^[11]

Exposure mappings

The next step after redefining the causal estimand of interest is to characterize the probability of spillover exposure for each subject in the analysis, given some vector of treatment assignment. Aronow and Samii (2017)^[12] present a method for obtaining a matrix of exposure probabilities for each unit in the analysis.

First, define a diagonal matrix with a vector of treatment assignment probabilities $\mathbf {P} =\operatorname {diag} \left(p_{\mathbf {z} _{1}},p_{\mathbf {z} _{2}},\dots ,p_{\mathbf {z} _{|\Omega |}}\right).$

Second, define an indicator matrix $\mathbf {I}$ of whether the unit is exposed to spillover or not. This is done by using an adjacency matrix as shown on the right, where information regarding a network can be transformed into an indicator matrix. This resulting indicator matrix will contain values of $d_{k}$ , the realized values of a random binary variable $D_{i}=f\left(\mathbf {Z} ,\theta _{i}\right)$ , indicating whether that unit has been exposed to spillover or not.

Third, obtain the sandwich product $\mathbf {I} _{k}\mathbf {P} \mathbf {I} _{k}^{\prime }$ , an N × N matrix which contains two elements: the individual probability of exposure $\pi _{i}\left(d_{k}\right)$ on the diagonal, and the joint exposure probabilities $\pi _{ij}\left(d_{k}\right)$ on the off diagonals:

\mathbf {I} _{k}\mathbf {P} \mathbf {I} _{k}^{\prime }=\left[{\begin{array}{cccc}{\pi _{1}(d_{k})}&\pi _{12}(d_{k})&\cdots &\pi _{1N}(d_{k})\\\pi _{21}(d_{k})&\pi _{2}(d_{k})&\cdots &\pi _{2N}(d_{k})\\\vdots &\vdots &\ddots &\\\pi _{N1}(d_{k})&\pi _{N2}(d_{k})&{}&\pi _{N}(d_{k})\end{array}}\right]

In a similar fashion, the joint probability of exposure of i being in exposure condition

d_{k}

and j being in a different exposure condition

d_{l}

can be obtained by calculating

\mathbf {I} _{k}\mathbf {P} \mathbf {I} _{l}^{\prime }

:

\mathbf {I} _{k}\mathbf {P} \mathbf {I} _{l}^{\prime }=\left[{\begin{array}{c c c c }{0}&{\pi _{12}\left(d_{k},d_{l}\right)}&{\dots }&{\pi _{1N}\left(d_{k},d_{l}\right)}\\{\pi _{21}\left(d_{k},d_{l}\right)}&{0}&{\ldots }&{\pi _{2N}\left(d_{k},d_{l}\right)}\\{\vdots }&{\vdots }&{\ddots }&{}\\\pi _{N1}(d_{k},d_{l})&\pi _{N2}(d_{k},d_{l})&&0\end{array}}\right]

Notice that the diagonals on the second matrix are 0 because a subject cannot be simultaneously exposed to two different exposure conditions at once, in the same way that a subject cannot reveal two different potential outcomes at once.

The obtained exposure probabilities $\pi$ then can be used for inverse probability weighting (IPW, described below), in an estimator such as the Horvitz–Thompson estimator.

One important caveat is that this procedure excludes all units whose probability of exposure is zero (ex. a unit that is not connected to any other units), since these numbers end up in the denominator of the IPW regression.

Need for inverse probability weights

Estimating spillover effects requires additional care: although treatment is directly assigned, spillover status is indirectly assigned and can lead to differential probabilities of spillover assignment for units. For example, a subject with 10 friend connections is more likely to be indirectly exposed to treatment as opposed to a subject with just one friend connection. Not accounting for varying probabilities of spillover exposure can bias estimates of the average spillover effect.

Figure 1 displays an example where units have varying probabilities of being assigned to the spillover condition. Subfigure A displays a network of 25 nodes where the units in green are eligible to receive treatment. Spillovers are defined as sharing at least one edge with a treated unit. For example, if node 16 is treated, nodes 11, 17, and 21 would be classified as spillover units. Suppose three of these six green units are selected randomly to be treated, so that ${\binom {6}{3}}=20$ different sets of treatment assignments are possible. In this case, subfigure B displays each node's probability of being assigned to the spillover condition. Node 3 is assigned to spillover in 95% of the randomizations because it shares edges with three units that are treated. This node will only be a control node in 5% of randomizations: that is, when the three treated nodes are 14, 16, and 18. Meanwhile, node 15 is assigned to spillover only 50% of the time—if node 14 is not directly treated, node 15 will not be assigned to spillover.

Using inverse probability weights

When analyzing experiments with varying probabilities of assignment, special precautions should be taken. These differences in assignment probabilities may be neutralized by inverse-probability-weighted (IPW) regression, where each observation is weighted by the inverse of its likelihood of being assigned to the treatment condition observed using the Horvitz-Thompson estimator.^[13] This approach addresses the bias that might arise if potential outcomes were systematically related to assignment probabilities. The downside of this estimator is that it may be fraught with sampling variability if some observations are accorded a high amount of weight (i.e. a unit with a low probability of being spillover is assigned to the spillover condition by chance).

Using randomization inference for hypothesis testing

In some settings, estimating the variability of a spillover effect creates additional difficulty. When the research study has a fixed unit of clustering, such as a school or household, researchers can use traditional standard error adjustment tools like cluster-robust standard errors, which allow for correlations in error terms within clusters but not across them.^[14] In other settings, however, there is no fixed unit of clustering. In order to conduct hypothesis testing in these settings, the use of randomization inference is recommended.^[15] This technique allows one to generate p-values and confidence intervals even when spillovers do not adhere to a fixed unit of clustering but nearby units tend to be assigned to similar spillover conditions, as in the case of fuzzy clustering.

Related Research Articles

In physics, the cross section is a measure of the probability that a specific process will take place when some kind of radiant excitation intersects a localized phenomenon. For example, the Rutherford cross-section is a measure of probability that an alpha particle will be deflected by a given angle during an interaction with an atomic nucleus. Cross section is typically denoted $σ$ (sigma) and is expressed in units of area, more specifically in barns. In a way, it can be thought of as the size of the object that the excitation must hit in order for the process to occur, but more exactly, it is a parameter of a stochastic process.

$<span class="mw-page-title-main">Diffraction</span> Phenomenon of the motion of waves$

Diffraction is the interference or bending of waves around the corners of an obstacle or through an aperture into the region of geometrical shadow of the obstacle/aperture. The diffracting object or aperture effectively becomes a secondary source of the propagating wave. Italian scientist Francesco Maria Grimaldi coined the word diffraction and was the first to record accurate observations of the phenomenon in 1660.

In physics, interference is a phenomenon in which two coherent waves are combined by adding their intensities or displacements with due consideration for their phase difference. The resultant wave may have greater intensity or lower amplitude if the two waves are in phase or out of phase, respectively. Interference effects can be observed with all types of waves, for example, light, radio, acoustic, surface water waves, gravity waves, or matter waves as well as in loudspeakers as electrical waves.

<span class="mw-page-title-main">Particle in a box</span> Physical model in quantum mechanics which is analytically solvable

In quantum mechanics, the particle in a box model describes a particle free to move in a small space surrounded by impenetrable barriers. The model is mainly used as a hypothetical example to illustrate the differences between classical and quantum systems. In classical systems, for example, a particle trapped inside a large box can move at any speed within the box and it is no more likely to be found at one position than another. However, when the well becomes very narrow, quantum effects become important. The particle may only occupy certain positive energy levels. Likewise, it can never have zero energy, meaning that the particle can never "sit still". Additionally, it is more likely to be found at certain positions than at others, depending on its energy level. The particle may never be detected at certain positions, known as spatial nodes.

In statistics, Gibbs sampling or a Gibbs sampler is a Markov chain Monte Carlo (MCMC) algorithm for obtaining a sequence of observations which are approximated from a specified multivariate probability distribution, when direct sampling is difficult. This sequence can be used to approximate the joint distribution ; to approximate the marginal distribution of one of the variables, or some subset of the variables ; or to compute an integral. Typically, some of the variables correspond to observations whose values are known, and hence do not need to be sampled.

In finance and economics, systematic risk is vulnerability to events which affect aggregate outcomes such as broad market returns, total economy-wide resource holdings, or aggregate income. In many contexts, events like earthquakes, epidemics and major weather catastrophes pose aggregate risks that affect not only the distribution but also the total amount of resources. That is why it is also known as contingent risk, unplanned risk or risk events. If every possible outcome of a stochastic economic process is characterized by the same aggregate result, the process then has no aggregate risk.

Field experiments are experiments carried out outside of laboratory settings.

In queueing theory, a discipline within the mathematical theory of probability, a Jackson network is a class of queueing network where the equilibrium distribution is particularly simple to compute as the network has a product-form solution. It was the first significant development in the theory of networks of queues, and generalising and applying the ideas of the theorem to search for similar product-form solutions in other networks has been the subject of much research, including ideas used in the development of the Internet. The networks were first identified by James R. Jackson and his paper was re-printed in the journal Management Science’s ‘Ten Most Influential Titles of Management Sciences First Fifty Years.’

Differential entropy is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy, a measure of average (surprisal) of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP). Differential entropy is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete entropy.

In condensed matter physics and crystallography, the static structure factor is a mathematical description of how a material scatters incident radiation. The structure factor is a critical tool in the interpretation of scattering patterns obtained in X-ray, electron and neutron diffraction experiments.

The Rubin causal model (RCM), also known as the Neyman–Rubin causal model, is an approach to the statistical analysis of cause and effect based on the framework of potential outcomes, named after Donald Rubin. The name "Rubin causal model" was first coined by Paul W. Holland. The potential outcomes framework was first proposed by Jerzy Neyman in his 1923 Master's thesis, though he discussed it only in the context of completely randomized experiments. Rubin extended it into a general framework for thinking about causation in both observational and experimental studies.

The theoretical and experimental justification for the Schrödinger equation motivates the discovery of the Schrödinger equation, the equation that describes the dynamics of nonrelativistic particles. The motivation uses photons, which are relativistic particles with dynamics described by Maxwell's equations, as an analogue for all types of particles.

In graph theory, a random geometric graph (RGG) is the mathematically simplest spatial network, namely an undirected graph constructed by randomly placing N nodes in some metric space and connecting two nodes by a link if and only if their distance is in a given range, e.g. smaller than a certain neighborhood radius, r.

The average treatment effect (ATE) is a measure used to compare treatments in randomized experiments, evaluation of policy interventions, and medical trials. The ATE measures the difference in mean (average) outcomes between units assigned to the treatment and units assigned to the control. In a randomized trial, the average treatment effect can be estimated from a sample using a comparison in mean outcomes for treated and untreated units. However, the ATE is generally understood as a causal parameter that a researcher desires to know, defined without reference to the study design or estimation procedure. Both observational studies and experimental study designs with random assignment may enable one to estimate an ATE in a variety of ways.

<span class="mw-page-title-main">Coulomb's law</span> Fundamental physical law of electromagnetism

Coulomb's inverse-square law, or simply Coulomb's law, is an experimental law of physics that calculates the amount of force between two electrically charged particles at rest. This electric force is conventionally called electrostatic force or Coulomb force. Although the law was known earlier, it was first published in 1785 by French physicist Charles-Augustin de Coulomb, hence the name. Coulomb's law was essential to the development of the theory of electromagnetism and maybe even its starting point, as it allowed meaningful discussions of the amount of electric charge in a particle.

In machine learning, the kernel embedding of distributions comprises a class of nonparametric methods in which a probability distribution is represented as an element of a reproducing kernel Hilbert space (RKHS). A generalization of the individual data-point feature mapping done in classical kernel methods, the embedding of distributions into infinite-dimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space operations such as inner products, distances, projections, linear transformations, and spectral analysis. This learning framework is very general and can be applied to distributions over any space $on which a sensible kernel function may be defined. For example, various kernels have been proposed for learning from data which are: vectors in, discrete classes/categories, strings, graphs/networks, images, time series, manifolds, dynamical systems, and other structured objects. The theory behind kernel embeddings of distributions has been primarily developed by Alex Smola, Le Song, Arthur Gretton, and Bernhard Schölkopf. A review of recent works on kernel embedding of distributions can be found in.$

<span class="mw-page-title-main">Up-and-Down Designs</span>

Up-and-down designs (UDDs) are a family of statistical experiment designs used in dose-finding experiments in science, engineering, and medical research. Dose-finding experiments have binary responses: each individual outcome can be described as one of two possible values, such as success vs. failure or toxic vs. non-toxic. Mathematically the binary responses are coded as 1 and 0. The goal of dose-finding experiments is to estimate the strength of treatment (i.e., the "dose") that would trigger the "1" response a pre-specified proportion of the time. This dose can be envisioned as a percentile of the distribution of response thresholds. An example where dose-finding is used is in an experiment to estimate the LD₅₀ of some toxic chemical with respect to mice.

In econometrics and related empirical fields, the local average treatment effect (LATE), also known as the complier average causal effect (CACE), is the effect of a treatment for subjects who comply with the experimental treatment assigned to their sample group. It is not to be confused with the average treatment effect (ATE), which includes compliers and non-compliers together. Compliance refers to the human-subject response to a proposed experimental treatment condition. Similar to the ATE, the LATE is calculated but does not include non-compliant parties. If the goal is to evaluate the effect of a treatment in ideal, compliant subjects, the LATE value will give a more precise estimate. However, it may lack external validity by ignoring the effect of non-compliance that is likely to occur in the real-world deployment of a treatment method. The LATE can be estimated by a ratio of the estimated intent-to-treat effect and the estimated proportion of compliers, or alternatively through an instrumental variable estimator.

The hyperbolastic functions, also known as hyperbolastic growth models, are mathematical functions that are used in medical statistical modeling. These models were originally developed to capture the growth dynamics of multicellular tumor spheres, and were introduced in 2005 by Mohammad Tabatabai, David Williams, and Zoran Bursac. The precision of hyperbolastic functions in modeling real world problems is somewhat due to their flexibility in their point of inflection. These functions can be used in a wide variety of modeling problems such as tumor growth, stem cell proliferation, pharma kinetics, cancer growth, sigmoid activation function in neural networks, and epidemiological disease progression or regression.

In computational and mathematical biology, a biological lattice-gas cellular automaton (BIO-LGCA) is a discrete model for moving and interacting biological agents, a type of cellular automaton. The BIO-LGCA is based on the lattice-gas cellular automaton (LGCA) model used in fluid dynamics. A BIO-LGCA model describes cells and other motile biological agents as point particles moving on a discrete lattice, thereby interacting with nearby particles. Contrary to classic cellular automaton models, particles in BIO-LGCA are defined by their position and velocity. This allows to model and analyze active fluids and collective migration mediated primarily through changes in momentum, rather than density. BIO-LGCA applications include cancer invasion and cancer progression.

References

↑ "Diffusion of Technologies within Social Networks: Evidence from a Coffee Training Program in Rwanda". IGC. 31 March 2010. Retrieved 2018-12-11.
↑ Zamora, Philippe; Rathelot, Roland; Gurgand, Marc; Duflo, Esther; Crépon, Bruno (2013-05-01). "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment". The Quarterly Journal of Economics. 128 (2): 531–580. doi:10.1093/qje/qjt001. hdl: 1721.1/82896 . ISSN 0033-5533. S2CID 15381050.
↑ "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities | Edward Miguel, Professor of Economics, University of California, Berkeley". emiguel.econ.berkeley.edu. Retrieved 2018-12-11.
↑ Avis, Eric; Ferraz, Claudio; Finan, Frederico (2018). "Do Government Audits Reduce Corruption? Estimating the Impacts of Exposing Corrupt Politicians" (PDF). Journal of Political Economy. 126 (5): 1912–1964. doi:10.1086/699209. hdl:10419/176135. S2CID 36161954.
↑ Weisburd, David; Telep, Cody W. (2014-05-01). "Hot Spots Policing: What We Know and What We Need to Know". Journal of Contemporary Criminal Justice. 30 (2): 200–220. doi:10.1177/1043986214525083. ISSN 1043-9862. S2CID 145692978.
↑ Sobel, Michael (2006). "What do randomized studies of housing mobility demonstrate?". Journal of the American Statistical Association. 101 (476): 1398–1407. doi:10.1198/016214506000000636. S2CID 739283.
↑ "PsycNET". psycnet.apa.org. Retrieved 2018-12-11.
↑ Manski, Charles F. (2013-02-01). "Identification of treatment response with social interactions" (PDF). The Econometrics Journal. 16 (1): S1–S23. doi:10.1111/j.1368-423X.2012.00368.x. hdl: 10419/64721 . ISSN 1368-4221. S2CID 1559596.
↑ Rosenbaum, Paul R. (2007). "Interference Between Units in Randomized Experiments". Journal of the American Statistical Association. 102 (477): 191–200. CiteSeerX 10.1.1.571.7817 . doi:10.1198/016214506000001112. S2CID 38153548.
↑ Rosenbaum, Paul R. (2007). "Interference Between Units in Randomized Experiments". Journal of the American Statistical Association. 102 (477): 191–200. CiteSeerX 10.1.1.571.7817 . doi:10.1198/016214506000001112. S2CID 38153548.
↑ Kao, Edward; Toulis, Panos (2013-02-13). "Estimation of Causal Peer Influence Effects". International Conference on Machine Learning: 1489–1497.
↑ Aronow, Peter M.; Samii, Cyrus (2017-12-01). "Estimating average causal effects under general interference, with application to a social network experiment". The Annals of Applied Statistics. 11 (4): 1912–1947. arXiv: 1305.6156 . doi:10.1214/16-aoas1005. ISSN 1932-6157. S2CID 26963450.
↑ Hortvitz, D. G.; Thompson, D. J. (1952). "A generalization of sampling without replacement from a finite universe". Journal of the American Statistical Association. 47 (260): 663–685. doi:10.1080/01621459.1952.10483446. JSTOR 2280784. S2CID 120274071.
↑ A. Colin Cameron; Douglas L. Miller. "A Practitioner's Guide to Cluster-Robust Inference" (PDF). Cameron.econ.ucdavis.edu. Retrieved 19 December 2018.
↑ "10 Things to Know About Randomization Inference". Egap.org. Retrieved 2018-12-11.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] "Diffusion of Technologies within Social Networks: Evidence from a Coffee Training Program in Rwanda". IGC. 31 March 2010. Retrieved 2018-12-11.

[2] Zamora, Philippe; Rathelot, Roland; Gurgand, Marc; Duflo, Esther; Crépon, Bruno (2013-05-01). "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment". The Quarterly Journal of Economics. 128 (2): 531–580. doi:10.1093/qje/qjt001. hdl: 1721.1/82896 . ISSN 0033-5533. S2CID 15381050.

[3] "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities | Edward Miguel, Professor of Economics, University of California, Berkeley". emiguel.econ.berkeley.edu. Retrieved 2018-12-11.

[4] Avis, Eric; Ferraz, Claudio; Finan, Frederico (2018). "Do Government Audits Reduce Corruption? Estimating the Impacts of Exposing Corrupt Politicians" (PDF). Journal of Political Economy. 126 (5): 1912–1964. doi:10.1086/699209. hdl:10419/176135. S2CID 36161954.

[5] Weisburd, David; Telep, Cody W. (2014-05-01). "Hot Spots Policing: What We Know and What We Need to Know". Journal of Contemporary Criminal Justice. 30 (2): 200–220. doi:10.1177/1043986214525083. ISSN 1043-9862. S2CID 145692978.

[6] Sobel, Michael (2006). "What do randomized studies of housing mobility demonstrate?". Journal of the American Statistical Association. 101 (476): 1398–1407. doi:10.1198/016214506000000636. S2CID 739283.

[7] "PsycNET". psycnet.apa.org. Retrieved 2018-12-11.

[8] Manski, Charles F. (2013-02-01). "Identification of treatment response with social interactions" (PDF). The Econometrics Journal. 16 (1): S1–S23. doi:10.1111/j.1368-423X.2012.00368.x. hdl: 10419/64721 . ISSN 1368-4221. S2CID 1559596.

[9] Rosenbaum, Paul R. (2007). "Interference Between Units in Randomized Experiments". Journal of the American Statistical Association. 102 (477): 191–200. CiteSeerX 10.1.1.571.7817 . doi:10.1198/016214506000001112. S2CID 38153548.

[10] Rosenbaum, Paul R. (2007). "Interference Between Units in Randomized Experiments". Journal of the American Statistical Association. 102 (477): 191–200. CiteSeerX 10.1.1.571.7817 . doi:10.1198/016214506000001112. S2CID 38153548.

[11] Kao, Edward; Toulis, Panos (2013-02-13). "Estimation of Causal Peer Influence Effects". International Conference on Machine Learning: 1489–1497.

[12] Aronow, Peter M.; Samii, Cyrus (2017-12-01). "Estimating average causal effects under general interference, with application to a social network experiment". The Annals of Applied Statistics. 11 (4): 1912–1947. arXiv: 1305.6156 . doi:10.1214/16-aoas1005. ISSN 1932-6157. S2CID 26963450.

[13] Hortvitz, D. G.; Thompson, D. J. (1952). "A generalization of sampling without replacement from a finite universe". Journal of the American Statistical Association. 47 (260): 663–685. doi:10.1080/01621459.1952.10483446. JSTOR 2280784. S2CID 120274071.

[14] A. Colin Cameron; Douglas L. Miller. "A Practitioner's Guide to Cluster-Robust Inference" (PDF). Cameron.econ.ucdavis.edu. Retrieved 19 December 2018.

[15] "10 Things to Know About Randomization Inference". Egap.org. Retrieved 2018-12-11.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]