Response rate (survey)

Last updated
A U.S. National Agricultural Statistics Service statistician explains response rate data at a 2017 briefing to clarify the context of crop production data. 20170810-OSEC-LSC-0151 (36510226215).jpg
A U.S. National Agricultural Statistics Service statistician explains response rate data at a 2017 briefing to clarify the context of crop production data.

In survey research, response rate, also known as completion rate or return rate, is the number of people who answered the survey divided by the number of people in the sample. It is usually expressed in the form of a percentage. The term is also used in direct marketing to refer to the number of people who responded to an offer.

Contents

The general consensus in academic surveys is to choose one of the six definitions summarized by the American Association for Public Opinion Research (AAPOR). [1] These definitions are endorsed by the National Research Council and the Journal of the American Medical Association, among other well recognized institutions.[ citation needed ] They are:

  1. Response Rate 1 (RR1) – or the minimum response rate, is the number of complete interviews divided by the number of interviews (complete plus partial) plus the number of non-interviews (refusal and break-off plus non-contacts plus others) plus all cases of unknown eligibility (unknown if housing unit, plus unknown, other).
  2. Response Rate 2 (RR2) – RR1 + counting partial interviews as respondents.
  3. Response Rate 3 (RR3) – estimates what proportion of cases of unknown eligibility is actually eligible. Those respondents estimated to be ineligible are excluded from the denominator. The method of estimation *must* be explicitly stated with RR3.
  4. Response Rate 4 (RR4) – allocates cases of unknown eligibility as in RR3, but also includes partial interviews as respondents as in RR2.
  5. Response Rate 5 (RR5) – is either a special case of RR3 in that it assumes that there are no eligible cases among the cases of unknown eligibility or the rare case in which there are no cases of unknown eligibility. RR5 is only appropriate when it is valid to assume that none of the unknown cases are eligible ones, or when there are no unknown cases.
  6. Response Rate 6 (RR6) – makes that same assumption as RR5 and also includes partial interviews as respondents. RR6 represents the maximum response rate.

The six AAPOR definitions vary with respect to whether or not the surveys are partially or entirely completed and how researchers deal with unknown nonrespondents. Definition #1, for example, does NOT include partially completed surveys in the numerator, while definition #2 does. Definitions 3–6 deal with the unknown eligibility of potential respondents who could not be contacted. For example, there is no answer at the doors of 10 houses you attempted to survey. Maybe 5 of those you already know house people who qualify for your survey based on neighbors telling you whom lived there, but the other 5 are completely unknown. Maybe the dwellers fit your target population, maybe they don't. This may or may not be considered in your response rate, depending on which definition you use.

Example: if 1,000 surveys were sent by mail, and 257 were successfully completed (entirely) and returned, then the response rate would be 25.7%.

Importance

A survey’s response rate is the result of dividing the number of people who were interviewed by the total number of people in the sample who were eligible to participate and should have been interviewed. [2] A low response rate can give rise to sampling bias if the nonresponse is unequal among the participants regarding exposure and/or outcome. Such bias is known as nonresponse bias.

For many years, a survey's response rate was viewed as an important indicator of survey quality. Many observers presumed that higher response rates assure more accurate survey results (Aday 1996; Babbie 1990; Backstrom and Hursh 1963; Rea and Parker 1997). But because measuring the relation between nonresponse and the accuracy of a survey statistic is complex and expensive, few rigorously designed studies provided empirical evidence to document the consequences of lower response rates until recently.

Such studies have finally been conducted in recent years, and several conclude that the expense of increasing the response rate frequently is not justified given the difference in survey accuracy.

One early example of a finding was reported by Visser, Krosnick, Marquette and Curtin (1996) who showed that surveys with lower response rates (near 20%) yielded more accurate measurements than did surveys with higher response rates (near 60 or 70%). [3] In another study, Keeter et al. (2006) compared results of a 5-day survey employing the Pew Research Center’s usual methodology (with a 25% response rate) with results from a more rigorous survey conducted over a much longer field period and achieving a higher response rate of 50%. In 77 out of 84 comparisons, the two surveys yielded results that were statistically indistinguishable. Among the items that manifested significant differences across the two surveys, the differences in proportions of people giving a particular answer ranged from 4 percentage points to 8 percentage points. [4]

A study by Curtin et al. (2000) tested the effect of lower response rates on estimates of the Index of Consumer Sentiment (ICS). They assessed the impact of excluding respondents who initially refused to cooperate (which reduces the response rate 5–10 percentage points), respondents who required more than five calls to complete the interview (reducing the response rate about 25 percentage points), and those who required more than two calls (a reduction of about 50 percentage points). They found no effect of excluding these respondent groups on estimates of the ICS using monthly samples of hundreds of respondents. For yearly estimates, based on thousands of respondents, the exclusion of people who required more calls (though not of initial refusers) had a very small one. [5]

Holbrook et al. (2005) assessed whether lower response rates are associated with less unweighted demographic representativeness of a sample. By examining the results of 81 national surveys with response rates varying from 5 percent to 54 percent, they found that surveys with much lower response rates decreased demographic representativeness within the range examined, but not by much. [6]

Choung et al. (2013) looked at community response rate to a mailed questionnaire about functional gastrointestinal disorders. The response rate to their community survey was 52%. Then, they took a random sample of 428 responders and 295 nonresponders for medical record abstraction, and compared nonresponders against responders. They found that respondents had a significantly higher body mass index and more health care seeking behavior for non-GI problems. However, except for diverticulosis and skin diseases, there was no significant difference between responders and nonresponders in terms of any gastrointestinal symptoms or specific medical diagnosis. [7]

Dvir and Gafni (2018) examined if consumer response rate is influenced by the amount of information provided. In a series of large-scale web experiments (n= 535 and n= 27,900), they compared variants of marketing web pages (also called Landing page), focusing on how changes to content amount impact users’ willingness to provide their e-mail address (a behavior called Conversion rate in marketing terms). The results showed significantly higher response rates on the shorter pages, which indicates that contrary to earlier work, not all response rate theories are effective online. [8]

Nevertheless, in spite of these recent research studies, a higher response rate is preferable because the missing data is not random. [9] There is no satisfactory statistical solution to deal with missing data that may not be random. Assuming an extreme bias in the responders is one suggested method of dealing with low survey response rates. A high response rate (>80%) from a small, random sample is preferable to a low response rate from a large sample. [10]

See also

Related Research Articles

In statistics, sampling bias is a bias in which a sample is collected in such a way that some members of the intended population have a lower or higher sampling probability than others. It results in a biased sample of a population in which all individuals, or instances, were not equally likely to have been selected. If this is not accounted for, results can be erroneously attributed to the phenomenon under study rather than to the method of sampling.

Observational error is the difference between a measured value of a quantity and its true value. In statistics, an error is not necessarily a "mistake". Variability is an inherent part of the results of measurements and of the measurement process.

In statistics, survey sampling describes the process of selecting a sample of elements from a target population to conduct a survey. The term "survey" may refer to many different types or techniques of observation. In survey sampling it most often involves a questionnaire used to measure the characteristics and/or attitudes of people. Different ways of contacting members of a sample once they have been selected is the subject of survey data collection. The purpose of sampling is to reduce the cost and/or the amount of work that it would take to survey the entire target population. A survey that measures the entire target population is called a census. A sample refers to a group or section of a population from which information is to be obtained

Sampling (statistics) Selection of data points in statistics.

In statistics, quality assurance, and survey methodology, sampling is the selection of a subset of individuals from within a statistical population to estimate characteristics of the whole population. Statisticians attempt to collect samples that are representative of the population in question. Sampling has lower costs and faster data collection than measuring the entire population and can provide insights in cases where it is infeasible to measure an entire population.

Sampling is the use of a subset of the population to represent the whole population or to inform about (social) processes that are meaningful beyond the particular cases, individuals or sites studied. Probability sampling, or random sampling, is a sampling technique in which the probability of getting any particular sample may be calculated. In cases where external validity is not of critical importance to the study's goals or purpose, researchers might prefer to use nonprobability sampling. Nonprobability sampling does not meet this criterion. Nonprobability sampling techniques are not intended to be used to infer from the sample to the general population in statistical terms. Instead, for example, grounded theory can be produced through iterative nonprobability sampling until theoretical saturation is reached.

Questionnaire construction refers to the design of a questionnaire to gather statistically useful information about a given topic. When properly constructed and responsibly administered, questionnaires can provide valuable data about any given subject.

Survey methodology is "the study of survey methods". As a field of applied statistics concentrating on human-research surveys, survey methodology studies the sampling of individual units from a population and associated techniques of survey data collection, such as questionnaire construction and methods for improving the number and accuracy of responses to surveys. Survey methodology targets instruments or procedures that ask one or more questions that may or may not be answered.

An opinion poll, often simply referred to as a poll or a survey, is a human research survey of public opinion from a particular sample. Opinion polls are usually designed to represent the opinions of a population by conducting a series of questions and then extrapolating generalities in ratio or within confidence intervals. A person who conducts polls is referred to as a pollster.

Questionnaire Series of questions for gathering information

A questionnaire is a research instrument that consists of a set of questions for the purpose of gathering information from respondents through survey or statistical study. A research questionnaire is typically a mix of close-ended questions and open-ended questions. Open-ended, long-term questions offer the respondent the ability to elaborate on their thoughts. The Research questionnaire was developed by the Statistical Society of London in 1838.

Response bias

Response bias is a general term for a wide range of tendencies for participants to respond inaccurately or falsely to questions. These biases are prevalent in research involving participant self-report, such as structured interviews or surveys. Response biases can have a large impact on the validity of questionnaires or surveys.

<i>Lancet</i> surveys of Iraq War casualties

The Lancet, one of the oldest scientific medical journals in the world, published two peer-reviewed studies on the effect of the 2003 invasion of Iraq and subsequent occupation on the Iraqi mortality rate. The first was published in 2004; the second in 2006. The studies estimate the number of excess deaths caused by the occupation, both direct and indirect.

In sociology and statistics research, snowball sampling is a nonprobability sampling technique where existing study subjects recruit future subjects from among their acquaintances. Thus the sample group is said to grow like a rolling snowball. As the sample builds up, enough data are gathered to be useful for research. This sampling technique is often used in hidden populations, such as drug users or sex workers, which are difficult for researchers to access. As sample members are not selected from a sampling frame, snowball samples are subject to numerous biases. For example, people who have many friends are more likely to be recruited into the sample. When virtual social networks are used, then this technique is called virtual snowball sampling.

In social science research, social-desirability bias is a type of response bias that is the tendency of survey respondents to answer questions in a manner that will be viewed favorably by others. It can take the form of over-reporting "good behavior" or under-reporting "bad", or undesirable behavior. The tendency poses a serious problem with conducting research with self-reports. This bias interferes with the interpretation of average tendencies as well as individual differences.

Acquiescence bias, also known as agreement bias, is a category of response bias common to survey research in which respondents have a tendency to select a positive response option or indicate a positive connotation disproportionately more frequently. Respondents do so without considering the content of the question or their 'true' preference. Acquiescence is sometimes referred to as "yea-saying" and is the tendency of a respondent to agree with a statement when in doubt. Questions affected by acquiescence bias take the following format: a stimulus in the form of a statement is presented, followed by 'agree/disagree,' 'yes/no' or 'true/false' response options. For example, a respondent might be presented with the statement "gardening makes me feel happy," and would then be expected to select either 'agree' or 'disagree.' Such question formats are favoured by both survey designers and respondents because they are straightforward to produce and respond to. The bias is particularly prevalent in the case of surveys or questionnaires that employ truisms as the stimuli, such as: "It is better to give than to receive" or "Never a lender nor a borrower be". Acquiescence bias can introduce systematic errors that affect the validity of research by confounding attitudes and behaviours with the general tendency to agree, which can result in misguided inference. Research suggests that the proportion of respondents who carry out this behaviour is between 10% and 20%.

Computer-assisted web interviewing (CAWI) is an Internet surveying technique in which the interviewee follows a script provided in a website. The questionnaires are made in a program for creating web interviews. The program allows for the questionnaire to contain pictures, audio and video clips, links to different web pages, etc. The website is able to customize the flow of the questionnaire based on the answers provided, as well as information already known about the participant. It is considered to be a cheaper way of surveying since one doesn't need to use people to hold surveys unlike computer-assisted telephone interviewing. With the increasing use of the Internet, online questionnaires have become a popular way of collecting information. The design of an online questionnaire has a dramatic effect on the quality of data gathered. There are many factors in designing an online questionnaire; guidelines, available question formats, administration, quality and ethic issues should be reviewed. Online questionnaires should be seen as a sub-set of a wider-range of online research methods.

Participation bias or non-response bias is a phenomenon in which the results of elections, studies, polls, etc. become non-representative because the participants disproportionately possess certain traits which affect the outcome. These traits mean the sample is systematically different from the target population, potentially resulting in biased estimates.

In statistics, non-sampling error is a catch-all term for the deviations of estimates from their true values that are not a function of the sample chosen, including various systematic errors and random errors that are not due to sampling. Non-sampling errors are much harder to quantify than sampling errors.

In survey sampling, total survey error includes all forms of survey error including sampling variability, interviewer effects, frame errors, response bias, and non-response bias. Total survey error is discussed in detail in many sources including Salant and Dillman.

With the application of probability sampling in the 1930s, surveys became a standard tool for empirical research in social sciences, marketing, and official statistics. The methods involved in survey data collection are any of a number of ways in which data can be collected for a statistical survey. These are methods that are used to collect information from a sample of individuals in a systematic way. First there was the change from traditional paper-and-pencil interviewing (PAPI) to computer-assisted interviewing (CAI). Now, face-to-face surveys (CAPI), telephone surveys (CATI), and mail surveys are increasingly replaced by web surveys.

Jon Krosnick

Jon Alexander Krosnick is a professor of Political Science, Communication, and Psychology, and director of the Political Psychology Research Group (PPRG) at Stanford University. Additionally, he is the Frederic O. Glover Professor in Humanities and Social Sciences and an affiliate of the Woods Institute for the Environment. Krosnick has served as a consultant for government agencies, universities, and businesses, has testified as an expert in court proceedings, and has been an on-air television commentator on election night.

References

  1. ↑ "Standard Definitions - AAPOR". Standard Definitions – AAPOR. AAPOR. Retrieved 3 March 2016.
  2. ↑ "Response Rates – An Overview." American Association for Public Opinion Research (AAPOR). 29 Sept 2008. http://www.aapor.org/Education-Resources/For-Researchers/Poll-Survey-FAQ/Response-Rates-An-Overview.aspx
  3. ↑ Visser, Penny S.; Krosnick, Jon A.; Marquette, Jesse; Curtin, Michael (1996). "Mail Surveys for Election Forecasting? An Evaluation of the Colombia Dispatch Poll". Public Opinion Quarterly. 60 (2): 181–227. doi:10.1086/297748.
  4. ↑ Keeter, Scott, Courtney Kennedy, Michael Dimock, Jonathan Best and Peyton Craighill. 2006. "Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey." Public Opinion Quarterly. 70(5): 759–779.
  5. ↑ Curtin, Richard; Presser, Stanley; Singer, Eleanor (2000). "The Effects of Response Rate Changes on the Index of Consumer Sentiment". Public Opinion Quarterly. 64 (4): 413–428. doi:10.1086/318638. PMID   11171024.
  6. ↑ Holbrook, Allyson, Jon Krosnick, and Alison Pfent. 2007. "The Causes and Consequences of Response Rates in Surveys by the News Media and Government Contractor Survey Research Firms." In Advances in telephone survey methodology, ed. James M. Lepkowski, N. Clyde Tucker, J. Michael Brick, Edith D. De Leeuw, Lilli Japec, Paul J. Lavrakas, Michael W. Link, and Roberta L. Sangster. New York: Wiley. https://pprg.stanford.edu/wp-content/uploads/2007-TSMII-chapter-proof.pdf
  7. ↑ Seon Choung, Rok; Richard Locke, III; Schleck, Cathy D.; Ziegenfuss, Jeanette Y.; Beebe, Timothy J.; Zinsmeister, Alan R.; Talley, Nicholas J. (2013). "A low response rate does not necessarily indicate non-response bias in gastroenterology survey research: a population-based study". Journal of Public Health. 21 (1): 87–95. doi:10.1007/s10389-012-0513-z. S2CID   38267579.
  8. ↑ Dvir, Nim; Gafni, Ruti (2018). "When Less Is More: Empirical Study of the Relation Between Consumer Behavior and Information Provision on Commercial Landing Pages". Informing Science: The International Journal of an Emerging Transdiscipline. 21: 019–039. doi: 10.28945/4015 . ISSN   1547-9684.
  9. ↑ Altman, DG; Bland, JM (Feb 2007). "Missing data". BMJ. 334 (7590): 424. doi:10.1136/bmj.38977.682025.2C. PMC   1804157 . PMID   17322261.
  10. ↑ Evans, SJ (Feb 1991). "Good surveys guide". BMJ. 302 (6772): 302–3. doi:10.1136/bmj.302.6772.302. PMC   1669002 . PMID   2001503.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.