Design effect

In survey methodology, the design effect (generally denoted as ${\displaystyle D_{\mathrm {eff} }}$ or ${\displaystyle D_{\mathrm {eft} }^{2}}$) is a measure of the expected impact of a sampling design on the variance of an estimator for some parameter. It is calculated as the ratio of the variance of an estimator based on a sample from an (often) complex sampling design, to the variance of an alternative estimator based on a simple random sample (SRS) of the same number of elements. ^{[1] : 258} The ${\displaystyle D_{\mathrm {eff} }}$ (be it estimated, or known a priori) can be used to adjust the variance of an estimator in cases where the sample is not drawn using simple random sampling. It may also be useful in sample size calculations and for quantifying the representativeness of a sample. The term "design effect" was coined by Leslie Kish in 1965.

The design effect is a positive real number that indicates an inflation (${\displaystyle D_{\mathrm {eff} }>1}$), or deflation (${\displaystyle D_{\mathrm {eff} }<1}$) in the variance of an estimator for some parameter, that is due to the study not using SRS (with ${\displaystyle D_{\mathrm {eff} }=1}$, when the variances are identical). ^{[2] : 53, 54}

Some potential complex sampling that could introduce ${\displaystyle D_{\mathrm {eff} }}$ that is different than 1 include: cluster sampling (such as when there is correlation between observations), stratified sampling, cluster randomized controlled trial, disproportional (unequal probability) sample, non-coverage, non-response, statistical adjustments of the data, etc..

${\displaystyle D_{\mathrm {eff} }}$ can be used in sample size calculations, quantifying the representative of a sample (to a target population), as well as for adjusting (often inflating) the variance of some estimator (in cases when we can calculate that estimator's variance assuming SRS). ^[3]

The term "Design effect" was coined by Leslie Kish in 1965. ^{[1] : 88, 258} Ever since, many calculations (and estimators) have been proposed, in the literature, for describing the effect of known sampling design on the increase/decrease in the variance of estimators of interest. In general, the design effect varies between statistics of interests, such as the total or ratio mean; it also matters if the design (e.g.: selection probabilities) are correlated with the outcome of interest. And lastly, it is influenced by the distribution of the outcome itself. All of these should be considered when estimating and using design effect in practice. ^{[4] : 13}

Definitions

${\displaystyle D_{\mathrm {eff} }}$

The design effect, commonly denoted by ${\displaystyle D_{\mathrm {eff} }}$ (at times with different subscripts), is the ratio of two theoretical variances for estimators of some parameter (${\displaystyle \theta }$): ^{[1] [5]}

• In the numerator is the actual variance for an estimator of some parameter (${\displaystyle {\hat {\theta }}_{w}}$) in a given sampling design ${\displaystyle p}$;
• In the denominator is the variance assuming the same sample size, but if the sample was obtained using the estimator we would use for a simple random sampling without replacement (${\displaystyle {\hat {\theta }}_{srswor}}$).

So that:

${\displaystyle D_{\mathrm {eff} ,p}({\hat {\theta }})={\frac {\mathrm {var} ({\hat {\theta }}_{w})}{\mathrm {var} ({\hat {\theta }}_{\mathrm {srswor} })}}}$

Put differently, ${\displaystyle D_{\mathrm {eff} }}$ is by how much more the variance had increased (or decreased, in some cases) because our sample was drawn and adjusted to a specific sampling design (e.g.: using weights, or other measures) as it would be if instead the sample was from a simple random sampling (without replacement). There are many ways of calculation ${\displaystyle D_{\mathrm {eff} }}$, depending on the parameter of interest (E.g.: population total, population mean, quantiles, ratio of quantities etc.), the estimator used, and the sampling design (e.g.: clustered sampling, stratified sampling, post-stratification, multi-stage sampling, etc.).

For estimating the population mean, the ${\displaystyle D_{\mathrm {eff} }}$ (for some sampling design p) is: ^{[4] : 4  [2] : 54}

${\displaystyle D_{\mathrm {eff} ,p}={\frac {\mathrm {var} _{p}({\bar {y}}_{p})}{(1-f)S_{y}^{2}/n}}}$

Where ${\displaystyle n}$ is the sample size, f is the fraction of the sample from the population ${\displaystyle (n/N)}$, ${\displaystyle 1-f}$ is the (squared) finite population correction (FPC), and ${\displaystyle S_{y}^{2}=}$ is the unbiassed sample variance.

The estimates of unit variance (or element variance) is when multiplying ${\displaystyle D_{\mathrm {eff} }}$ by the element's variance, so to incorporate all the complexities of the sample design. ^{[1] : 259}

Notice how the definition of ${\displaystyle D_{\mathrm {eff} }}$ is based on parameters of the population that we often do not know (i.e.: the variances of estimators under two different sampling designs). The process of estimating ${\displaystyle D_{\mathrm {eff} }}$ for specific designs will be described in the following section. ^{[6] : 98}

A general formula for the (theoretical) design effect of estimating a total (not the mean), for some design, is given in Cochran 1977. ^{[2] : 54}

${\displaystyle D_{\mathrm {eft} }}$

A related quantity to ${\displaystyle D_{\mathrm {eff} }}$, proposed by Kish in 1995, is called ${\displaystyle D_{\mathrm {eft} }}$ (Design Effect Factor). ^{[7] : 56  [4]} It is defined on the square root of the variance ratios, and also the denominator uses a simple random sample with replacement (srswr), instead of without replacement (srswor):

${\displaystyle D_{\mathrm {eft} }={\sqrt {\frac {\mathrm {var} ({\hat {\theta }}_{w})}{\mathrm {var} ({\hat {\theta }}_{\mathrm {srswr} })}}}}$

In this later definition (proposed in 1995, vs 1965) it was argued that srs "without replacement" (with its positive effect on the variance) should be captured in the definition of the design effect, since it is part of the sampling design. It is also more directly related to the use in inference (since we often use +Z*DE*SE, not +Z*DE*VAR when creating confidence intervals). Also since the finite population correction (FPC) is also harder to compute in some situations. But for many cases when the population is very large, Deft is (almost) the square root of ${\displaystyle D_{\mathrm {eff} }}$ (${\displaystyle D_{\mathrm {eft} }\approx {\sqrt {D_{\mathrm {eff} }}}}$).

The original intention for ${\displaystyle D_{\mathrm {eft} }}$ was to have it "express the effects of sample design beyond the elemental variability ${\displaystyle {\frac {S_{m}^{2}}{m}}}$, removing both the unit of measurement and sample size as nuisance parameters", this is done in order to make the design effect generalizable (relevant for) many statistics and variables within the same survey (and even between surveys). ^{[7] : 55} However, followup works have shown that the calculation of design effect, for parameters such as a population total or mean, has dependence on the variability of the outcome measure, which limits the original aspiration of Kish for this measure. However, this statement may loosely (i.e.: under some conditions) be true for the weighted mean. ^{[4] : 5}

Effective sample size

The effective sample size, also defined by Kish in 1965, is the original sample size divided by the design effect. ^{[1] : 162, 259  [8] : 190, 192} This quantity reflects what would be the sample size that is needed to achieve the current variance of the estimator (for some parameter) with the existing design, if the sample design (and its relevant parameter estimator) were based on a simple random sample. ^[9]

Namely:

${\displaystyle n_{\text{eff}}={\frac {n}{D_{eff}}}}$

Put differently, it says how many responses we are left with when using an estimator that correctly adjusts for the design effect of the sampling design. For example, using the weighted mean with inverse probability weighting, instead of the simple mean.

It is also possible to get the effective sample size ratio by taking the inverse of ${\displaystyle D_{\mathrm {eff} }}$ (i.e.: ${\displaystyle {\frac {n_{eff}}{n}}={\frac {1}{D_{eff}}}}$).

When using Kish's design effect for unequal weights, you may use the following simplified formula for "Kish's Effective Sample Size" ^{[10] [1] : 162, 259}

${\displaystyle n_{\text{eff}}={\frac {n}{D_{\text{eff}}}}={\frac {n}{\frac {\overline {w^{2}}}{{\overline {w}}^{2}}}}={\frac {n}{\frac {{\frac {1}{n}}\sum _{i=1}^{n}w_{i}^{2}}{\left({\frac {1}{n}}\sum _{i=1}^{n}w_{i}\right)^{2}}}}={\frac {n}{\frac {n\sum _{i=1}^{n}w_{i}^{2}}{(\sum _{i=1}^{n}w_{i})^{2}}}}={\frac {(\sum _{i=1}^{n}w_{i})^{2}}{\sum _{i=1}^{n}w_{i}^{2}}}}$

Design effect for well-known sampling designs

Sampling design dictates how design effect should be calculated

Different sampling designs differ substantially in their impact on estimators (such as the mean) in terms of their bias and variance.

For example, in the cluster sampling case the units may have equal or unequal selection probabilities, irrespective to their intra-class correlation (and their negative effect of increasing the variance of our estimators). In the case of stratified sampling, the probabilities may be equal (EPSEM) or unequal. But regardless, the usage of the prior information on the stratum size in the population, during the sampling stage, could yield statistical efficiency of our estimators. For example: if we know that gender is correlated with our outcome of interest, and also know that the male-female ratio for some population is 50%-50%. Then if we made sure to sample exactly half of each gender, we've thus reduced the variance of the estimators because we've removed the variability caused by unequal proportion of males-females in our sample. Lastly, in case of adjusting to non-coverage, non-response or some stratum split of the population (unavailable during the sampling stage), we may use statistical procedures (E.g.: post-stratification and others). The result of such procedures may lead to estimations of the sampling probabilities that are similar, or very different, than the true sampling probabilities of the units. The quality of these estimators depends on the quality of the auxiliary information and the missing at random assumptions used in creating them. Even when these sampling probability estimators (propensity scores) manage to capture most of the phenomena that has produced them - the impact of the variable selection probabilities on the estimators may be small or large, depending on the data (details in the next section).

Due to the large variety in sampling designs (with or without an effect on unequal selection probabilities), different formulas have been developed to capture the potential design effect, as well as to estimate the correct variance of estimators. Sometimes, these different design effects can be compounded together (as in the case of unequal selection probability and cluster sampling, more details in the following sections). Whether or not to use these formulas, or just assume SRS, depends on to expected amount of bias reduced vs the increase in estimator variance (and in the overhead of methodological and technical complexity). ^{[1] : 426}

Unequal selection probabilities

Sources for unequal selection probabilities

There are various ways to sample units so that each unit would have the exact same probability of selection. Such methods are called equal probability sampling (EPSEM) methods. Some of the more basic methods include simple random sample (SRS, either with or without replacement) and systematic sampling for getting a fixed sample size. There is also Bernoulli sampling with a random sample size. More advanced techniques such as stratified sampling and cluster sampling can also be designed to be EPSEM. For example, in cluster sampling we can make sure to sample each cluster with probability that is proportional to its size, and then measure all the units inside the cluster. A more complex method for cluster sampling is to use a two-stage sampling by which we sample clusters at the first stage (as before, proportional to cluster size), and sample from each cluster at the second stage using SRS with a fixed proportion (E.g.: sample half of the cluster). ^{[11] : 3–8}

In their works, Kish and others highlights several known reasons that lead to unequal selection probabilities: ^{[1] : 425  [8] : 185  [7] : 69  [12] : 50, 395  [13] : 306}

1. Disproportional sampling due to selection frame or procedure. This happens when a researcher purposefully designs their sample so to over/under sample specific sub-populations or clusters. There are many cases in which this might happen. For example:

   • In stratified sampling when units from some strata are known to have a larger variance than other strata. In such cases, the intention of the researcher may be to use this prior knowledge about the variance between stratum in order to reduce the overall variance of an estimator of some population level parameter of interest (e.g.: the mean). This can be achieved by a strategy known as optimum allocation , in which a strata ${\displaystyle h}$ is over sampled proportional to higher standard deviation and lower sampling cost (i.e.: ${\displaystyle f_{h}\propto {\frac {S_{h}}{\sqrt {C_{h}}}}}$, where ${\displaystyle S_{h}}$ is the standard deviation of the outcome in ${\displaystyle h}$, and ${\displaystyle C_{h}}$ relates to the cost of recruiting one element from ${\displaystyle h}$). An example of an optimum allocation is Neyman's optimal allocation which, when cost is fixed for recruiting each stratum, the sample size is: ${\displaystyle n_{h}=n{\frac {W_{h}S_{Uh}}{\sum _{h}W_{h}S_{Uh}}}}$. Where the summation is over all strata; n is the total sample size; ${\displaystyle n_{h}}$ is the sample size for stratum h; ${\displaystyle W_{h}={\frac {N_{h}}{N}}}$ the relative size of stratum h as compared to the entire population N; and ${\displaystyle S_{Uh}}$ is the standard error of in stratum h. A related concept to optimum design is optimal experimental design.
   • If there is interest in comparing two stratum (E.g.: people from two specific socio-demographic groups, or from two regions, etc.), in which case the smaller group may be over-sampled. This way, the variance of the estimator that compares the two groups is reduced.
   • In cluster sampling there may be clusters of different sizes but the procedure samples from all clusters using SRS, and all elements in the cluster are measured (for example, if the cluster sizes are not known upfront at the stage of sampling).
   • When using a two-stage sampling so that in the first stage the clusters are sampled proportionally to their size (a.k.a.: PPS Probability Proportional to Size), but then at the second stage only a specific fixed number of units (E.g.: one or two) are selected from each cluster - this may happen due to convenience/budget considerations. A similar case is when the first stage attempt to sample using PPS, but the number of elements in each unit are inaccurate (so that some smaller cluster may have a higher-than-it-should chance of being selected. And vis-versa for larger clusters with too-small of a chance to be sampled). In such cases, the larger the errors in the sampling frame in the first stage - the larger will be the needed unequal selection probabilities. ^{[6] : 109}
   • When the frame used for sampling includes duplication of some of the items, thus leading some items to have a larger probability than others to be sampled (E.g.: if the sampling frame was created by merging several lists. Or if recruiting users from several ad channels - in which some of the users are available for recruitment from several of the channels, while others are available to be recruited from only one of the channels). In each of these cases - different units would have different sampling probability, thus making this sampling procedure to not be EPSEM. ^{[11] : 3–8  [8] : 186}
   • When several different samples/frames are combined. For example, if running different ad campaigns for recruiting respondents. Or when combining results from several studies done by different researchers and/or at different times (i.e.: Meta-analysis). ^{[8] : 188}

   When disproportional sampling happens, due to sampling design decisions, the researcher may (sometimes) be able to traceback the decision and accurately calculate the exact inclusion probability. When these selection probabilities are hard to traceback, they may be estimated using some propensity score model combined with information from auxiliary variables (E.g.: age, gender, etc.).

2. Non-coverage. ^{[1] : 527, 528} This happens, for example, if people are sampled based on some pre-defined list that doesn't include all the people in the population (E.g.: a phone book or using ads to recruit people to a survey). These missing units are missing due to some failure of creating the sampling frame, as opposed to deliberate exclusion of some people (E.g.: minors, people who cannot vote, etc.). The effect of non-coverage on sampling probability is considered difficult to measure (and adjust for) in various survey situations, unless strong assumptions are made.
3. Non-response. This refers to the failure of obtaining measurements on sampled units that are intended to be measured. Reasons for non-response are varied and depends on the context. A person may be temporarily unavailable, for example if they are not available to pick up the phone when the survey is done. A person may also refuse to answer the survey due to a variety of reasons, E.g.: different tendencies of people from different ethnic/demographic/socio-economic groups to respond in general; insufficient incentive to spend the time or share data; the identity of the institution that is running the survey; inability to respond (E.g.: due to illness, illiteracy, or a language barrier); respondent is not found (E.g.: they've moved an appartement); the response was lost/destroyed during encoding or transmission (i.e.: measurement error). In the context of surveys, these reasons may be related to answering the entire survey or just specific questions. ^{[1] : 532  [8] : 186}
4. statistical adjustments. These may include methods such as post-stratification, raking, or propensity score (estimation) models - used to perform an ad-hoc adjustment of the sample to some known (or estimated) stratum sizes. Such procedures are used to mitigate issues in the sampling ranging from sampling error, under-coverage of the sampling frame to non-response. ^{[14] : 45  [15]} For example, if a simple random sample is used, a post-stratification (using some auxiliary information) does not offer an estimator that is uniformly better than just an unweighted estimator. However, it can be viewed as a more "robust" estimator. ^[16] Alternatively, these methods can be used to make the sample more similar to some target "controls" (i.e.: population of interest), a process also known as "standardization". ^{[8] : 187} In such cases, these adjustments help with providing unbiased estimators (often with the cost of increased variance, as seen in the following sections). If the original sample is a nonprobability sampling, then post-stratification adjustments are just similar to an ad-hoc quota sampling. ^{[8] : 188, 189}

When the sampling design is fully known (leading to some ${\displaystyle p_{h}}$ probability of selection for some element from strata h), and the non-response is measurable (i.e.: we know that only ${\displaystyle r_{h}}$ observations answered in strata h), then an exactly known inverse probability weight can be calculated for each element i from strata h using:${\displaystyle w_{i}={\frac {1}{p_{h}r_{h}}}}$. ^{[8] : 186} Sometimes a statistical adjustment, such as post-stratification or raking, is used for estimating the selection probability. E.g.: when comparing the sample we have with same target population, also known as matching to controls. The estimation process may be focused only on adjusting the existing population to an alternative population (for example, if trying to extrapolate from a panel drawn from several regions to an entire country). In such a case, the adjust might be focused on some calibration factor ${\displaystyle c_{i}}$ and the weights be calculated as ${\displaystyle w_{i}={\frac {c_{i}}{p_{h}r_{h}}}}$. ^{[8] : 187} However, in other cases, both the under-coverage and non-response are all modeled in one go as part of the statistical adjustment, which leads to an estimation of the overall sampling probability (let's say ${\displaystyle p_{i}'}$). In such a case, the weights are simply: ${\displaystyle w_{i}={\frac {1}{p_{i}'}}}$. Notice that when statistical adjustments are used, ${\displaystyle w_{i}}$ is often estimated based on some model. The formulation in the following sections assume this ${\displaystyle w_{i}}$ is known, which is not true for statistical adjustments (since we only have ${\displaystyle {\widehat {w}}_{i}}$). However, if it is assumed that the estimation error of ${\displaystyle {\widehat {w}}_{i}}$ is very small then the following sections can be used as if it was known. Having this assumption be true depends on the size of the sample used for modeling, and is worth keeping in mind during analysis.

When the selection probabilities may be different, the sample size is random, and the pairwise selection probabilities are independent, we call this Poisson sampling. ^[17]

"Design based" vs "model based" for describing properties of estimators

When adjusting for unequal probability selection through "individual case weights" (E.g.: inverse probability weighting), we get various types of estimators for quantities of interest. Estimators such as Horvitz–Thompson estimator yield unbiased estimators (if the selection probabilities are indeed known, or approximately known), for total and the mean of the population. Deville and Särndal (1992) coined the term "calibration estimator" for estimators using weights such that they satisfy some condition, such as having the sum of weights equal the population size. And more generally, that the weighted sum of weights is equal some quantity of an auxiliary variable: ${\displaystyle \sum w_{i}x_{i}=X}$ (e.g.: that the sum of weighted ages of the respondents is equal to the population size in each age bucket). ^{[18] [15] : 132  [19] : 1}

The two primary ways to argue about the properties of calibration estimators are: ^{[15] : 133–134  [20]}

1. randomization based (or, sampling design based) - in these cases, the weights (${\displaystyle w_{i}}$) and values of the outcome of interest ${\displaystyle y_{i}}$ that are measured in the sample are all treated as known. In this framework, there is variability in the (known) values of the outcome (Y). However, the only randomness comes from which of the elements in the population were picked into the sample (often denoted as ${\displaystyle I_{i}}$, getting 1 if element ${\displaystyle i}$ is in the sample and 0 if it is not). For a simple random sample, each ${\displaystyle I_{i}}$ will be an i.i.d bernoulli distribution with some parameter ${\displaystyle p}$. For general EPSEM (equal probability sampling) ${\displaystyle I_{i}}$ will still be bernoulli with some parameter ${\displaystyle p}$, but they will no longer be independent random variables. For something like post stratification, the number of elements at each strata can be modeled as a multinomial distribution with different ${\displaystyle p_{h}}$ inclusion probabilities for each element belonging to some strata ${\displaystyle h}$. In these cases the sample size itself can be a random variable.
2. model based - in these cases the sample is fixed, the weights are fixed, but the outcome of interest is treated as a random variable. For example, in the case of post-stratification, the outcome can be modeled as some linear regression function where the independent variables are indicator variables mapping each observation to its relevant strata, and the variability comes with the error term.

As we will see later, some proofs in the literature rely on the randomization-based framework, while others focus on the model-based perspective. When moving from the mean to the weighted mean, more complexity is added. For example, in the context of survey methodology oftentimes the population size itself is considered an unknown quantity that is estimated. So in the calculation of the weighted mean is in fact based on a ratio estimator, with an estimator of the total at the numerator and an estimator of the population size in the denominator (making the variance calculation to be more complex). ^[21]

Common types of weights

There are many types (and subtypes) of weights, with different ways to use and interpret them. With some weights their absolute value has some important meaning, while with other weights the important part is the relative values of the weights to each other. This section presents some of the more common types of weights so that they can be referenced in followup sections.

• Frequency weights are a basic type of weighting, presented in introduction to statistics courses. With these, each weight is an integer number that indicates the absolute frequency of an item in the sample. These are also sometimes termed repeat (or occurrence) weights. The specific value has an absolute meaning that is lost if the weights are transformed (e.g.: scaling). For example: if we have the numbers 10 and 20 with the frequency weights values of 2 and 3, then when "spreading" our data it is: 10,10, 20, 20, 20 (with weights of 1 to each of these items). Frequency weights includes the amount of information contained in a dataset, and thus allows things like creating unbiased weighted variance estimation using Bessel's correction. Notice that such weights are often random variables, since the specific number of items we will see from each value in the dataset is random.
• inverse-variance weighting is when each element is assigned a weight that is the inverse of its (known) variance. ^{[22] [8] : 187} When all elements have the same expectancy, using such weights for calculating weighted average has the least variance among all weighted averages. In the common formulation, these weights are known and not random (this seems related to reliability weights^{[ definition needed ]}).
• Normalized (convex) weights is a set of weights that form a convex combination. I.e.: each weight is a number between 0 and 1, and the sum of all weights is equal to 1. Any set of (non negative) weights can be turned into normalized weights by dividing each weight with the sum of all weights, making these weights normalized to sum to 1.

A related form are weights normalized to sum to sample size (n). These (non-negative) weights sum to the sample size (n), and their mean is 1. Any set of weights can be normalized to sample size by dividing each weight with the average of all weights. These weights have a nice relative interpretation where elements with weight larger than 1 are more "important" (in terms of their relative influence on, say, the weighted mean) then the average observation, while weights smaller than 1 are less "important" than the average observation.

• Inverse probability weighting is when each element is given a weight that is (proportional) to the inverse probability of selecting that element. E.g., by using ${\displaystyle w_{i}={\frac {1}{p_{i}}}}$. ^{[8] : 185} With inverse probability weights, we learn how many items each element "represents" in the target population. Hence, the sum of such weights returns the size of the target population of interest. Inverse probability weights can be normalized to sum to 1 or normalized to sum to the sample size (n), and many of the calculations from the following sections will yield the same results.

When a sample is EPSEM then all the probabilities are equal and the inverse of the selection probability yield weights that are all equal to one another (they are all equal to ${\displaystyle {\frac {N}{n}}={\frac {1}{f}}}$, where ${\displaystyle n}$ is the sample size and ${\displaystyle N}$ is the population size). Such a sample is called a self weighting sample. ^{[8] : 193}

There are also indirect ways of applying "weighted" adjustments. For example, the existing cases may be duplicated to impute missing observations (e.g.: from non-response), with variance estimated using methods such as multiple imputation. A complementary dealing of data is to remove (give weight of 0) to some cases. For example, when wanting to reduce the influence of over-sampled groups that are less essential for some analysis. Both cases are similar in nature to inverse probability weighting but the application in practice gives more/less rows of data (making the input potentially simpler to use in some software implementation), instead of applying an extra column of weights. Nevertheless, the consequences of such implementations are similar to just using weights. So while in the case of removing observations the data can easily be handled by common software implementations, the case of adding rows requires special adjustments for the uncertainty estimations. Not doing so may lead to erroneous conclusions(i.e.: there is no free lunch when using alternative representation of the underlying issues). ^{[8] : 189, 190}

The term "Haphazard weights", coined by Kish, is used to refer to weights that correspond to unequal selection probabilities, but ones that are not related to the expectancy or variance of the selected elements. ^{[8] : 190, 191}

Haphazard weights with estimated ratio-mean (${\displaystyle {\hat {\bar {Y}}}}$) - Kish's design effect

Formula

When taking an unrestricted sample of ${\displaystyle n}$ elements, we can then randomly split these elements into ${\displaystyle H}$ disjoint stratum, each of them containing some size of ${\displaystyle n_{h}}$ elements so that ${\displaystyle \sum \limits _{h=1}^{H}n_{h}=n}$. All elements in each strata ${\displaystyle h}$ has some (known) non-negative weight assigned to them (${\displaystyle w_{h}}$). The weight ${\displaystyle w_{h}}$ can be produced by the inverse of some unequal selection probability for elements in each strata ${\displaystyle h}$ (i.e.: inverse probability weighting following something like post-stratification). In this setting, Kish's design effect, for the increase in variance of the sample weighted mean due to this design (reflected in the weights), versus SRS of some outcome variable y (when there is no correlation between the weights and the outcome, i.e.: haphazard weights) is: ^{[1] : 427  [8] : 191(4.2)}

${\displaystyle D_{eff}={\frac {n\sum \limits _{h=1}^{H}(n_{h}w_{h}^{2})}{(\sum \limits _{h=1}^{H}n_{h}w_{h})^{2}}}}$

By treating each item from coming from its own stratum ${\displaystyle \forall h:n_{h}=1}$, Kish (in 1992) simplified the above formula to the (well known) following version: ^{[8] : 191(4.3)  [23] : 318  [4] : 8}

${\displaystyle D_{eff}={\frac {n\sum _{i=1}^{n}w_{i}^{2}}{(\sum _{i=1}^{n}w_{i})^{2}}}={\frac {{\frac {1}{n}}\sum _{i=1}^{n}w_{i}^{2}}{\left({\frac {1}{n}}\sum _{i=1}^{n}w_{i}\right)^{2}}}={\frac {\overline {w^{2}}}{{\overline {w}}^{2}}}}$

This version of the formula is valid when one stratum had several observations taken from it (i.e.: each having the same weight), or when there are just many stratum were each one had one observation taken from it, but several of them had the same probability of selection. While the interpretation is slightly different, the calculation of the two scenarios comes out to be the same.

Notice that Kish's definition of the design effect is closely tied to the coefficient of variation (also termed relative variance, relvariance or relvar for short) of the weights (when using uncorrected (population level) sample standard deviation for estimation). This has several notations in the literature: ^{[8] : 191  [12] : 396}

${\displaystyle D_{eff}=1+L=1+{C_{V}}^{2}=1+relvar(w)=1+{\frac {V(w)}{{\bar {w}}^{2}}}}$.

Where ${\displaystyle V(w)={\frac {\sum (w_{i}-{\bar {w}})^{2}}{n}}}$ is the population variance of ${\displaystyle w}$, and ${\displaystyle {\bar {w}}={\frac {\sum w_{i}}{n}}}$ is the mean. When the weights are normalized to sample size (so that their sum is equal to n and their mean is equal to 1), then ${\displaystyle {C_{V}}^{2}=V(w)}$ and the formula reduces to ${\displaystyle D_{eff}=1+V(w)}$. While it is true we assume the weights are fixed, we can think of their variance as the variance of an empirical distribution defined by sampling (with equal probability) one weight from our set of weights (similar to how we would think about the correlation of x and y in a simple linear regression).

[Proof]

${\displaystyle {C_{V}}^{2}=\left({\frac {s_{w}}{\bar {w}}}\right)^{2}={\frac {\frac {\sum _{i=1}^{n}(w_{i}-{\bar {w}})^{2}}{n}}{{\bar {w}}^{2}}}={\frac {\frac {\sum _{i=1}^{n}{w_{i}}^{2}-n{\bar {w}}^{2}}{n}}{{\bar {w}}^{2}}}={\frac {{\overline {w}}^{2}-{\bar {w}}^{2}}{{\bar {w}}^{2}}}={\frac {{\overline {w}}^{2}}{{\bar {w}}^{2}}}-1=D_{eff}-1\implies D_{eff}=1+{C_{V}}^{2}}$

Assumptions and proofs

The above formula gives the increase in the variance of the weighted mean based on "haphazard" weights, which reflects when y are observations selected using unequal selection probabilities (with no within-cluster correlation, and no relationship to the expectancy or variance of the outcome measurement); ^{[8] : 190, 191} and y' are the observations we would have had if we got them from simple random sample, then:

${\displaystyle D_{eff(kish)}={\frac {var\left({\bar {y}}_{w}\right)}{var\left({\bar {y}}'\right)}}={\frac {var\left({\frac {\sum \limits _{i=1}^{n}w_{i}y_{i}}{\sum \limits _{i=1}^{n}w_{i}}}\right)}{var\left({\frac {\sum \limits _{i=1}^{n}y_{i}'}{n}}\right)}}}$

From a model based perspective, ^[24] this formula holds when all n observations (${\displaystyle y_{1},...,y_{n}}$) are (at least approximately) uncorrelated (${\displaystyle \forall (i\neq j):cor(y_{i},y_{j})=0}$), with the same variance (${\displaystyle \sigma ^{2}}$) in the response variable of interest (y). It also assumes the weights themselves are not a random variable but rather some known constants (E.g.: the inverse of probability of selection, for some pre-determined and known sampling design).

[Proof]

The following is a simplified proof for when there are no clusters (i.e.: no Intraclass correlation between element of the sample) and each strata includes only one observation: ^[24]

${\displaystyle {\begin{aligned}var\left({\bar {y}}_{w}\right)&{\stackrel {1}{=}}var\left({\frac {\sum \limits _{i=1}^{n}w_{i}y_{i}}{\sum \limits _{i=1}^{n}w_{i}}}\right){\stackrel {2}{=}}var\left(\sum \limits _{i=1}^{n}w_{i}'y_{i}\right){\stackrel {3}{=}}\sum \limits _{i=1}^{n}var\left(w_{i}'y_{i}\right)\\&{\stackrel {4}{=}}\sum \limits _{i=1}^{n}w_{i}'^{2}var\left(y_{i}\right){\stackrel {5}{=}}\sum \limits _{i=1}^{n}w_{i}'^{2}\sigma ^{2}{\stackrel {6}{=}}\sigma ^{2}\sum \limits _{i=1}^{n}w_{i}'^{2}{\stackrel {7}{=}}\sigma ^{2}{\frac {\sum \limits _{i=1}^{n}w_{i}^{2}}{\left(\sum \limits _{i=1}^{n}w_{i}\right)^{2}}}\\&{\stackrel {8}{=}}\sigma ^{2}{\frac {\sum \limits _{i=1}^{n}w_{i}^{2}}{\left(\sum \limits _{i=1}^{n}w_{i}{\frac {n}{n}}\right)^{2}}}{\stackrel {9}{=}}\sigma ^{2}{\frac {\sum \limits _{i=1}^{n}w_{i}^{2}}{\left({\frac {\sum \limits _{i=1}^{n}w_{i}}{n}}\right)^{2}n^{2}}}{\stackrel {10}{=}}{\frac {\sigma ^{2}}{n}}{\frac {\frac {\sum \limits _{i=1}^{n}w_{i}^{2}}{n}}{\left({\frac {\sum \limits _{i=1}^{n}w_{i}}{n}}\right)^{2}}}{\stackrel {11}{=}}{\frac {\sigma ^{2}}{n}}{\frac {\overline {w^{2}}}{{\bar {w}}^{2}}}{\stackrel {12}{=}}var\left({\bar {y}}'\right)D_{eff}\\&\implies D_{eff(kish)}={\frac {var\left({\bar {y}}_{w}\right)}{var\left({\bar {y}}'\right)}}\\\end{aligned}}}$

Transitions:

1. from definition of the weighted mean.
2. using normalized (convex) weights definition (weights that sum to 1): ${\displaystyle w_{i}'={\frac {w_{i}}{\sum \limits _{i=1}^{n}w_{i}}}}$.
3. sum of uncorrelated random variables.
4. If the weights are constants (from the basic properties of the variance). Another way to say it is that the weights are known upfront for each observation i. Namely that we are actually calculating ${\displaystyle var\left({\bar {y}}_{w}|w\right)}$
5. when all observations have the same variance (${\displaystyle \sigma ^{2}}$).

The conditions on y are trivially held if the y observations are i.i.d with the same expectation and variance. In such case we have ${\displaystyle y=y'}$, and we can estimate ${\displaystyle var\left({\bar {y}}_{w}\right)}$ by using ${\displaystyle {\overline {var\left({\bar {y}}_{w}\right)}}={\overline {var\left({\bar {y}}\right)}}\times D_{eff}}$. ^{[8] [25]} If the y's are not all with the same expectations then we cannot use the estimated variance for calculation, since that estimation assumes that all ${\displaystyle y_{i}}$s have the same expectation. Specifically, if there is a correlation between the weights and the outcome variable y, then it means that the expectation of y is not the same for all observations (but rather, dependent on the specific weight value for each observation). In such a case, while the design effect formula might still be correct (if the other conditions are met), it would require a different estimator for the variance of the weighted mean. For example, it might be better to use a weighted variance estimator.

If different ${\displaystyle y_{i}}$s have different variances, then while the weighted variance could capture the correct population-level variance, the Kish's formula for the design effect may no longer be true.

A similar issue happens if there is some correlation structure in the samples (such as when using cluster sampling).

Alternative definitions in the literature

It is worth noting that some sources in the literature give the following alternative definition to Kish's design effect, stating it is: "the ratio of the variance of the weighted survey mean under disproportionate stratified sampling to the variance under proportionate stratified sampling when all stratum unit variances are equal". ^{[23] : 318  [12] : 396}

This definition can be slightly misleading, since it might be interpreted to mean that "proportionate stratified sampling" was achieved via stratified sampling, in which a pre-determined number of units is selected from each stratum. Such selection will yield reduced variance (as compared with simple random sample), since it removes some of the uncertainty in the specific number of elements per stratum. This is different than Kish's original definition which compared the variance of the design to a simple random sample (which would yield approximately probability proportionate to sample, but not exactly - due to the variance in sample sizes in each stratum). Park and Lee (2006) reflects on this by stating that "The rationale behind the above derivation is that the loss in precision of [the weighted mean] due to haphazard unequal weighting can be approximated by the ratio of the variance under disproportionate stratified sampling to that under the proportionate stratified sampling". ^{[4] : 8} How far are these two definitions differ from each other is not mentioned in the literature.^{[ citation needed ]} In his book from 1977, Cochran provides a formula for the proportional increase in variance due to deviation from optimum allocation (what, it Kish's formulas, would be called L). ^{[2] : 116} However, the connection from that formula to Kish's L is not apparent.^{[ citation needed ]}

Alternative naming conventions

Earlier papers would use the term ${\displaystyle Deff}$. ^{[8] : 192} As more definitions of design effect appeared, Kish's design effect for unequal selection probabilities was denoted ${\displaystyle Deff_{kish}}$ (or ${\displaystyle Deft_{kish}^{2}}$) or simply ${\displaystyle deff_{K}}$ for short. ^{[4] : 8  [12] : 396  [23] : 318} Kish's design effect is also known as the "Unequal Weighting Effect" (or just UWE), termed by Liu et al. in 2002. ^{[26] : 2124}

When the outcome correlates with the selection probabilities

Spencer's Deff for estimated total (${\displaystyle {\hat {Y}}}$)

The estimator for the total is the "p-expanded with replacement" estimator (a.k.a.: pwr-estimator or Hansen and Hurwitz). It is based on a simple random sample (with replacement, denoted SIR) of m items (${\displaystyle y_{k}}$) from a population of size M. Each item has a probability of ${\displaystyle p_{k}}$ (k from 1 to N) to be drawn in a single draw (${\displaystyle \sum _{U}p_{k}=1}$, i.e.: it's a multinomial distribution). The probability that a specific ${\displaystyle y_{k}}$ will appear in our sample is ${\displaystyle p_{k}}$. The "p-expanded with replacement" value is ${\displaystyle Z_{i}={\frac {y_{k}}{p_{k}}}}$ with the following expectancy: ${\displaystyle E[Z_{i}]=E[I_{i}{\frac {y_{k}}{p_{k}}}]={\frac {y_{k}}{p_{k}}}E[I_{i}]={\frac {y_{k}}{p_{k}}}p_{k}=y_{k}}$. Hence ${\displaystyle {\hat {Y}}_{pwr}={\frac {1}{m}}\sum _{i}^{m}Z_{i}}$, the pwr-estimator, is an unbiased estimator for the sum total of y. ^{[2] : 51}

In 2000, Bruce D. Spencer proposed a formula for estimating the design effect for the variance of estimating the total (not the mean) of some quantity (${\displaystyle {\hat {Y}}}$), when there is correlation between the selection probabilities of the elements and the outcome variable of interest. ^[27]

In this setup, a sample of size n is drawn (with replacement) from a population of size N. Each item is drawn with probability ${\displaystyle P_{i}}$ (where ${\displaystyle \sum _{i=1}^{N}P_{i}=1}$, i.e.: multinomial distribution). The selection probabilities are used to define the Normalized (convex) weights: ${\displaystyle w_{i}={\frac {1}{nP_{i}}}}$. Notice that for some random set of n items, the sum of weights will be equal to 1 only by expectancy (${\displaystyle E[w_{i}]=1}$) with some variability of the sum around it (i.e.: the sum of elements from a poisson binomial distribution). The relationship between ${\displaystyle y_{i}}$ and ${\displaystyle P_{i}}$ is defined by the following (population) simple linear regression:

${\displaystyle y_{i}=\alpha +\beta P_{i}+\epsilon _{i}}$

Where ${\displaystyle y_{i}}$ is the outcome of element i, which linearly depends on ${\displaystyle P_{i}}$ with the intercept ${\displaystyle \alpha }$ and slope ${\displaystyle \beta }$. The residual from the fitted line is ${\displaystyle \epsilon _{i}=y_{i}-(\alpha +\beta P_{i})}$. We can also define the population variances of the outcome and the residuals as ${\displaystyle \sigma _{y}^{2}}$ and ${\displaystyle \sigma _{\epsilon }^{2}}$. The correlation between ${\displaystyle P_{i}}$ and ${\displaystyle y_{i}}$ is ${\displaystyle \rho _{y,P}}$.

Spencer's (approximate) design effect, for estimating the total of y, is: ^{[27] : 138  [28] : 4  [12] : 401}

${\displaystyle Deff_{Spencer}=(1-{\hat {\rho }}_{y,P}^{2})(1+L)+\left({\frac {\hat {\alpha }}{{\hat {\sigma }}_{y}}}\right)^{2}L}$

Where:

• ${\displaystyle {\hat {\rho }}_{y,P}^{2}}$ estimates ${\displaystyle \rho _{y,P}^{2}}$
• ${\displaystyle {\hat {\alpha }}}$ estimates the slope ${\displaystyle \alpha }$
• ${\displaystyle {\hat {\sigma }}_{y}}$ estimates the population variance ${\displaystyle \sigma _{y}}$, and
• L is the relative variance of the weights, as defined in Kish's formula: : ${\displaystyle L=cv_{w}^{2}=relvar(w)={\frac {V(w)}{{\bar {w}}^{2}}}}$.

This assumes that the regression model fits well so that the probability of selection and the residuals are independent, since it leads to the residuals, and the square residuals, to be uncorrelated with the weights. I.e.: that ${\displaystyle \rho _{\epsilon ,W}=0}$ and also ${\displaystyle \rho _{\epsilon ^{2},W}=0}$. ^{[27] : 138}

When the population size (N) is very large, the formula can be written as: ^{[23] : 319}

${\displaystyle Deff_{Spencer}=(1-{\hat {\rho }}_{y,P}^{2})(1+cv_{w}^{2})+\left({\frac {1}{cv_{Y}^{2}}}\right)^{2}cv_{w}^{2}}$

(since ${\displaystyle \alpha ={\bar {Y}}-\beta \times {\bar {P}}={\bar {Y}}-\beta \times {\frac {1}{N}}\approx {\bar {Y}}}$, where ${\displaystyle cv_{Y}^{2}={\frac {\sigma _{Y}^{2}}{\bar {Y}}}}$)

This approximation assumes that the linear relationship between P and y holds. And also that the correlation of the weights with the errors, and the errors squared, are both zero. I.e.: ${\displaystyle \rho _{w,e}=0}$ and ${\displaystyle \rho _{w,e^{2}}=0}$. ^{[28] : 4}

We notice that if ${\displaystyle {\hat {\rho }}_{y,P}\approx 0}$, then ${\displaystyle {\hat {\alpha }}\approx {\bar {y}}}$ (i.e.: the average of y). In such a case, the formula reduces to

${\displaystyle Deff_{Spencer}=(1+L)+\left({\frac {1}{relvar(y)}}\right)^{2}L}$

Only if the variance of y is much larger than its mean then the right-most term is close to 0 (i.e.: ${\displaystyle relvar(y)={\frac {\sigma _{y}}{\bar {Y}}}\approx 0}$), which reduces Spencer's design effect (for the estimatoed total) to be equal to Kish's design effect (for the ratio means): ^{[28] : 5}${\displaystyle Deff_{Spencer}\approx (1+L)=Deff_{Kish}}$. Otherwise, the two formula's will yield different results, which demonstrates the difference between the design effect of the total vs the one of the mean.

Park and Lee's Deff for estimated ratio-mean (${\displaystyle {\hat {\bar {Y}}}}$)

In 2001, Park and Lee extended Spencer's formula to the case of the ratio-mean (i.e.: estimating the mean by dividing the estimator of the total with the estimator of the population size). It is: ^{[28] : 4}

${\displaystyle Deff_{Park\&Lee}=(1-{\hat {\rho }}_{y,P}^{2})(1+cv_{w}^{2})+{\frac {{\hat {\rho }}_{y,P}^{2}}{cv_{P}^{2}}}cv_{w}^{2}}$

Where:

• ${\displaystyle cv_{P}^{2}}$ is the (estimated) coefficient of variation of the probabilities of selection.

Park and Lee's formula is exactly equal to Kish's formula when ${\displaystyle {\hat {\rho }}_{y,P}^{2}=0}$. Both formula's relate to the design effect of the mean of y (while Spencer's Deff relates to the estimation of the total).

In general, the Deff for the total (${\displaystyle {\hat {Y}}}$) tends to be less efficient than the Deff for the ratio mean (${\displaystyle {\hat {\bar {Y}}}}$) when ${\displaystyle \rho _{y,P}}$ is small. And in general, ${\displaystyle \rho _{y,P}}$ impacts the efficiency of both design effects. ^{[4] : 8}

Cluster sampling

For data collected using cluster sampling we assume the following structure:

• ${\displaystyle n_{k}}$ observations in each cluster and K clusters, and with a total of ${\displaystyle n=\sum n_{k}}$ observations.
• The observations have a block correlation matrix in which every pair of observations from the same cluster is correlated with an intra-class correlation of ${\displaystyle \rho }$, while every pair from difference clusters are uncorrelated. ^[29] I.e., for every pair of observations, ${\displaystyle i}$ and ${\displaystyle j}$, if they belong to the same cluster ${\displaystyle k}$, we get ${\displaystyle cov(y_{i},y_{j})=\rho \sigma ^{2}}$. And two items from two different clusters are not correlated, i.e.: ${\displaystyle cov(y_{i},y_{j})=0}$.
• An element from any cluster is assumed to have the same variance: ${\displaystyle var(y_{i})=\sigma _{h}^{2}=\sigma ^{2}}$.

When clusters are all of the same size ${\displaystyle n^{*}}$, the design effect D_eff, proposed by Kish in 1965 (and later re-visited by others), is given by: ^{[1] : 162  [12] : 399  [4] : 9  [30] [31] [13] : 241}

${\displaystyle D_{\text{eff}}=1+(n^{*}-1)\rho .}$

It is sometimes also denoted as ${\displaystyle Deff_{C}}$. ^{[26] : 2124}

In various papers, when cluster sizes are not equal, the above formula is also used with ${\displaystyle n^{*}}$ as the average cluster size (it is also sometimes denoted as ${\displaystyle {\bar {b}}}$). ^{[32] [24] : 105} In such cases, Kish's formula (using the average cluster weight) serves as a conservative (upper bound) of the exact design effect. ^{[24] : 106}

Alternative formulas exists for unequal cluster sizes. ^{[1] : 193} Followup work had discussed the sensitivity of using the average cluster size with various assumptions. ^[33]

Unequal selection probabilities ${\displaystyle \times }$ Cluster sampling

In his paper from 1987, Kish proposed a combined design effect that incorporates both the effects due to weighting that accounts for unequal selection probabilities as well as cluster sampling: ^{[32] [24] : 105  [34] : 4  [28] : 2}

${\displaystyle Deff_{Kish}={\frac {n\sum \limits _{h=1}^{H}(n_{h}w_{h}^{2})}{\sum \limits _{h=1}^{H}(n_{h}w_{h})^{2}}}\left(1+(n^{*}-1)\rho \right)=deff_{k}\times deff_{C}}$

With notations similar to above.

This formula received a model based justification, proposed in 1999 by Gabler et al. ^[24]

Stratified sampling ${\displaystyle \times }$ unequal selection probabilities ${\displaystyle \times }$ Cluster sampling

In 2000, Liu and Aragon proposed a decomposition of unequal selection probabilities design effect for different strata in stratified sampling. ^[35] In 2002, Liu et al. extended that work to account for stratified sample were within each strata is a set of unequal selection probability weights. The cluster sampling is either global or per strata. ^[26] Similar work was done also by Park et al. in 2003. ^[36]

Uses

Deff is primarily used for several purposes: ^{[13] : 85}

• When developing the design - to evaluate its efficiency. I.e.: if there is potentially "too much" increase in variance due to some decision, or if the new design is more efficient (e.g.: as in stratified sampling).
• As a way for guiding sample size (overall, per stratum, per cluster, etc.), and also
• When evaluating potential problems with a post-hoc weighting analysis (E.g.: from non-response adjustments). ^[6] There is no universal rule-of-thumb for which design effect value is "too high", but the literature indicates that ${\displaystyle Deff>1.5}$ is likely to lead to some attention. ^{[12] : 396}

In his 1995 paper, Kish proposed the following categorization of when Deff is, and is not, useful: ^{[7] : 57–62}

• Design effect is unnecessary when: the source population is closely i.i.d, or when the sample design of the data was drawn as a simple random sample. It is also less useful when the sample size is relatively small (at least partially, for practical reasons). And also if only descriptive statistics are of interest (i.e.: point estimation). It is also suggested that if standard errors are needed for only a handful of statistics, it may be o.k. to ignore Deff.
• Design effect is necessary when: averaging sampling errors for different variables measured on the same survey. OR when averaging the same measured quantity from several surveys over a period of time. Or when extrapolating from the error of simple statistics (e.g.: the mean) to more complex ones (e.g.: regression coefficients). When designing a future survey (but with proper caution). As an aiding statistic to identify glaring issues with the data or its analysis (e.g.: ranging from mistakes to the presence of Outliers). ^{[8] : 191}

When planning the sample size, work has been done to correct the design effect so to separates the interviewer effect (measurement error) from the effects of the sampling design on the sampling variance. ^[37]

While Kish originally hoped to have the design effect be able to be agnostic as possible to the underlying distribution of the data, sampling probabilities, their correlations, and the statistics of interest - followup research has shown that these do influence the design effect. Hence, careful attention to these properties should be taken into account when deciding which Deff calculation to use, and how to use it. ^{[4] : 13  [28] : 6}

Software implementations

Kish's design effect is implemented in various statistical software:

• R: surveysummary from the survey package.
• Python: design_effect from the balance package. ^[38]

History

The term "Design effect" was introduced by Leslie Kish in 1965 in his book "Survey Sampling". ^{[1] : 88, 258} In his paper from 1995, ^{[7] : 73} Kish mentions that a similar concept, termed "Lexis ratio", was described at the end of the 19th century. The closely related Intraclass correlation was described by Fisher in 1950, while computations of ratios of variances were already published by Kish and others from the late 40s to the 50s. One of the precursors for Kish's definition was the work done by Cornfield in 1951. ^{[39] [4]}

In his original book from 1965, Kish proposed the general definition for the design effect (ratio of variances of two estimators, one from a sample with some design and the other from a simple random sample). In his book, Kish proposed the formula for the design effect of cluster sampling (with intraclass correlation); ^{[1] : 162} as well as the famous design effect formula for unequal probability sampling. ^{[1] : 427} These are often known as "Kish's design effect", and have been merged later into a single formula.

See also

• Variance inflation factor (VIF). VIF and Deff are similar concepts in that they are both ratios of variances of estimating some parameter under alternative models.
• Effective sample size

References

1. Kish, Leslie (1965). Survey Sampling. New York: John Wiley & Sons, Inc. ISBN   0-471-10949-5.
2. Carl-Erik Sarndal; Bengt Swensson; Jan Wretman (1992). Model Assisted Survey Sampling. Springer. ISBN   978-0-387-97528-3.
3. Heo, Moonseong; Kim, Yongman; Xue, Xiaonan; Kim, Mimi Y. (2010). "Sample size requirement to detect an intervention effect at the end of follow-up in a longitudinal cluster randomized trial". Statistics in Medicine. 29 (3): 382–390. doi:10.1002/sim.3806. PMID   20014353. S2CID   30001378. Archived from the original on 5 January 2013.
4. Park, Inho, and Hyunshik Lee. "Design effects for the weighted mean and total estimators under complex survey sampling." Quality control and applied statistics 51.4 (2006): 381–384 (based on google scholar). Vol. 30, No. 2, pp. 183-193. Statistics Canada, Catalogue No. 12-001. Survey Methodology December 2004 (based on the PDF) (pdf)
5. Everitt, B.S. (2002) The Cambridge Dictionary of Statistics, 2nd Edition. CUP. ISBN   0-521-81099-X
6. Kalton, G., J. M. Brick, and T. Le. "Estimating components of design effects for use in sample design. In household sample surveys in developing and transition countries,(Sales No. E. 05. XVII. 6). Department of Economic and Social Affairs." Statistics Division, United Nations, New York (2005). (pdf)
7. Kish, Leslie. "Methods for design effects." Journal of official Statistics 11.1 (1995): 55 (pdf)
8. Kish, Leslie, and J. Official Stat. "Weighting for unequal Pi." (1992): 183–200. (pdf link)
9. Tom Leinster (18 December 2014). "Effective Sample Size".
10. "Design Effects and Effective Sample Size".
11. Source: Frerichs, R.R. Rapid Surveys (unpublished), © 2004. N, chapter 4 - Equal Probability of Selection (pdf)
12. Valliant, Richard, Jill A. Dever, and Frauke Kreuter. Practical tools for designing and weighting survey samples. New York: Springer, 2013.
13. Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Nashville, TN: John Wiley & Sons. ISBN   978-0-471-16240-7
14. Dever, Jill A., and Richard Valliant. "A comparison of variance estimators for post-stratification to estimated control totals." Survey Methodology 36.1 (2010): 45-56. (pdf)
15. Kott, Phillip S. "Using calibration weighting to adjust for nonresponse and coverage errors." Survey Methodology 32.2 (2006): 133. (pdf)
16. Holt, David, and TM Fred Smith. "Post stratification." Journal of the Royal Statistical Society, Series A (General) 142.1 (1979): 33-46. (pdf)
17. Ghosh, Dhiren, and Andrew Vogt. "Sampling methods related to Bernoulli and Poisson Sampling." Proceedings of the Joint Statistical Meetings. American Statistical Association Alexandria, VA, 2002. (pdf)
18. Deville, Jean-Claude, and Carl-Erik Särndal. "Calibration estimators in survey sampling." Journal of the American statistical Association 87.418 (1992): 376-382.
19. Brick, J. Michael, Jill Montaquila, and Shelley Roth. "Identifying problems with raking estimators." annual meeting of the American Statistical Association, San Francisco, CA. 2003. (pdf)
20. Keiding, Niels, and David Clayton. "Standardization and control for confounding in observational studies: a historical perspective." Statistical Science (2014): 529-558. (pdf)
21. Thomas Lumley (https://stats.stackexchange.com/users/249135/thomas-lumley), How to estimate the (approximate) variance of the weighted mean?, URL (version: 2021-05-25): link
22. Kalton, Graham. "Standardization: A technique to control for extraneous variables." Journal of the Royal Statistical Society, Series C (Applied Statistics) 17.2 (1968): 118-136.
23. Henry, Kimberly A., and Richard Valliant. "A design effect measure for calibration weighting in single-stage samples." Survey Methodology 41.2 (2015): 315-331. (pdf)
24. Gabler, Siegfried, Sabine Häder, and Partha Lahiri. "A model based justification of Kish's formula for design effects for weighting and clustering." Survey Methodology 25 (1999): 105–106. (pdf)
25. Little, Roderick J., and Sonya Vartivarian. "Does weighting for nonresponse increase the variance of survey means?." Survey Methodology 31.2 (2005): 161. pdf link
26. Liu, Jun, Vince Iannacchione, and Margie Byron. "Decomposing design effects for stratified sampling." Proceedings of the survey research methods section, american statistical association. 2002. (pdf)
27. Spencer, Bruce D. "An approximate design effect for unequal weighting when measurements may correlate with selection probabilities." Survey Methodology 26 (2000): 137-138. (pdf)
28. Park, Inho, and Hyunshik Lee. "The design effect: do we know all about it." Proceedings of the Annual Meeting of the American Statistical Association. 2001. (pdf)
29. Alexander K. Rowe; Marcel Lama; Faustin Onikpo; Michael S. Deming (2002). "Design effects and intraclass correlation coefficients from a health facility cluster survey in Benin". International Journal for Quality in Health Care. 14 (6): 521–523. doi: 10.1093/intqhc/14.6.521 . PMID   12515339.
30. Bland, M (2005), "Cluster randomised trials in the medical literature", Notes for talks, York Univ
31. Methods in Sample Surveys Archived 27 November 2019 at the Wayback Machine (pages 5–6)
32. Kish, L. (1987). Weighting in ${\displaystyle Deft^{2}}$. The Survey Statistician, June 1987. (this paper doesn't seem to be available online, but is references in several places as the original source of this formula)
33. Lynn, Peter, and Siegfried Gabler. Approximations to b* in the prediction of design effects due to clustering. No. 2004-07. ISER Working Paper Series, 2004. (pdf)
34. Gabler, Siegfried, Sabine Hader, and Peter Lynn. Design effects for multiple design samples. No. 2005-12. ISER Working Paper Series, 2005. (pdf)
35. Liu, J., and E. Aragon. "Subsampling strategies in longitudinal surveys." Proceedings of the Survey Research Methods Section, American Statistical Association. 2000. (pdf)
36. Park, Inho (2003). "Design effects and survey planning" (PDF).
37. Zins, Stefan, and Jan Pablo Burgard. "Considering interviewer and design effects when planning sample sizes." SURVEY METHODOLOGY 46.1 (2020): 93-119. (paper - html)
38. Sarig T, Galili T, Eilat R. balance--a Python package for balancing biased data samples. arXiv preprint arXiv:2307.06024. 2023 Jul 12.
39. Cochran, William G. "Modern methods in the sampling of human populations." American journal of public health and the nation's health 41.6 (1951): 647–668.

Further reading

• The Impact of Typical Survey Weighting Adjustments on the Design Effect: A Case Study