Survey response effects

Last updated

Survey response effects are variations in survey responses that result from seemingly inconsequential aspects of survey design and administration. Susceptibility to these effects varies depending on the stability of one's beliefs. Those without a strong attitude on an issue, for instance, would more be more prone to survey response effects than those strongly for or against the issue. [1] These effects can be broadly grouped as consistency or contrast effects. Consistency effects are effects that lead to survey responses that agree, not to be confused with the identically-named term used to refer to the phenomenon in which respondents intentionally try to get their survey responses to agree one another. Contrast effects on the other hand, lead to opposing responses. [2]

Contents

Common survey response effects

Significance

In general, people answer survey questions with remarkable inconsistency, that is their responses are unstable. Response instability is when people are asked the same questions in repeated surveys and respond with conflicting answers. To explain this instability, John Zaller and Stanley Feldman argue that how people respond to surveys depends on which schemas, or considerations, are most readily available in the mind. They claim that an attitude about a given issue, at a given time, is a reflection of an average of the considerations in the mind at that time. [1] To apply a metaphor, attitude is climate; considerations are weather features. And since the pool of considerations in one's mind is constantly changing, attitudes are largely unstable over time. Due to this instability, survey response effects can significantly influence the outcome of surveys and should be considered when using surveys to inform policy making.

Contradictory theories of survey response instability

Philip Converse has attributed survey response instability to respondents lacking meaningful beliefs, [1] [7] while others have chalked it up to measurement errors and vague language in surveys. [1] [8]

See also

Related Research Articles

Appeal to emotion or argumentum ad passiones is a logical fallacy characterized by the manipulation of the recipient's emotions in order to win an argument, especially in the absence of factual evidence. This kind of appeal to emotion is a type of red herring and encompasses several logical fallacies, including appeal to consequences, appeal to fear, appeal to flattery, appeal to pity, appeal to ridicule, appeal to spite, and wishful thinking.

Questionnaire construction refers to the design of a questionnaire to gather statistically useful information about a given topic. When properly constructed and responsibly administered, questionnaires can provide valuable data about any given subject.

A field of applied statistics of human research surveys, survey methodology studies the sampling of individual units from a population and associated techniques of survey data collection, such as questionnaire construction and methods for improving the number and accuracy of responses to surveys. Survey methodology includes instruments or procedures that ask one or more questions that may or may not be answered.

Opinion poll type of survey

An opinion poll, often simply referred to as a poll or a survey, is a human research survey of public opinion from a particular sample. Opinion polls are usually designed to represent the opinions of a population by conducting a series of questions and then extrapolating generalities in ratio or within confidence intervals.

Agenda-setting theory describes the "ability to influence the importance placed on the topics of the public agenda". Agenda-setting theory was formally developed by Max McCombs and Donald Shaw in a study on the 1968 American presidential election. Agenda setting is a social science theory; it also attempts to make predictions. The theory also suggests that media has a great influence to their audience by instilling what they should think instead of what they think. That is, if a news item is covered frequently and prominently, the audience will regard the issue as more important.

Questionnaire Research instrument consisting of a series of questions and other prompts for the purpose of gathering information from respondents

A questionnaire is a research instrument consisting of a series of questions for the purpose of gathering information from respondents. The questionnaire was invented by the Statistical Society of London in 1838.

Response bias a general term for a wide range of tendencies for survey participants to respond inaccurately or falsely to questions

Response bias is a general term for a wide range of tendencies for participants to respond inaccurately or falsely to questions. These biases are prevalent in research involving participant self-report, such as structured interviews or surveys. Response biases can have a large impact on the validity of questionnaires or surveys.

Salience is the state or condition of being prominent. The Oxford English Dictionary defines salience as "most noticeable or important." The concept is discussed in communication, semiotics, linguistics, sociology, psychology, and political science. It has been studied with respect to interpersonal communication, persuasion, politics, and its influence on mass media.

The adaptive unconscious, first coined by social psychologist Daniel Wegner in 2002, is described as a set of mental processes that is able to affect judgement and decision-making, but is out of reach of the conscious mind. It is thought to be adaptive as it helps to keep the organism alive. Architecturally, the adaptive unconscious is said to be unreachable because it is buried in an unknown part of the brain. This type of thinking evolved earlier than the conscious mind, enabling the mind to transform information and think in ways that enhance an organism's survival. It can be described as a quick sizing up of the world which interprets information and decides how to act very quickly and outside the conscious view. The adaptive unconscious is active in everyday activities such as learning new material, detecting patterns, and filtering information. It is also characterized by being unconscious, unintentional, uncontrollable, and efficient without requiring cognitive tools. Lacking the need for cognitive tools does not make the adaptive unconscious any less useful than the conscious mind as the adaptive unconscious allows for processes like memory formation, physical balancing, language, learning, and some emotional and personalities processes that includes judgement, decision making, impression formation, evaluations, and goal pursuing. Despite being useful, the series of processes of the adaptive unconscious will not always result in accurate or correct decisions by the organism. The adaptive unconscious is affected by things like emotional reaction, estimations, and experience and is thus inclined to stereotyping and schema which can lead to inaccuracy in decision making. The adaptive conscious does however help decision making to eliminate cognitive biases such as prejudice because of its lack of cognitive tools.

A self-report study is a type of survey, questionnaire, or poll in which respondents read the question and select a response by themselves without interference. A self-report is any method which involves asking a participant about their feelings, attitudes, beliefs and so on. Examples of self-reports are questionnaires and interviews; self-reports are often used as a way of gaining participants' responses in observational studies and experiments.

<i>The Nature and Origins of Mass Opinion</i> book by John Zaller

The Nature and Origins of Mass Opinion is a 1992 non-fiction book by political scientist John Zaller that examines the processes by which individuals form and express political opinions and the implications this has for public opinion research. The book has been called "the single most important book on public opinion since V. O. Key's 1961 classic, Public Opinion and American Democracy."

The priming theory states that media images stimulate related thoughts in the minds of audience members.

In the social sciences, framing comprises a set of concepts and theoretical perspectives on how individuals, groups, and societies, organize, perceive, and communicate about reality.

John A. Bargh is a social psychologist currently working at Yale University, where he has formed the Automaticity in Cognition, Motivation, and Evaluation (ACME) Laboratory. Bargh's work focuses on automaticity and unconscious processing as a method to better understand social behavior, as well as philosophical topics such as free will. Much of Bargh's work investigates whether behaviors thought to be under volitional control may result from automatic interpretations of and reactions to external stimuli, such as words.

Norbert Schwarz is Provost Professor in the Department of Psychology and the USC Marshall School of Business at the University of Southern California and a co-director of the USC Dornsife Mind and Society Center.

Implicit attitudes are evaluations that occur without conscious awareness towards an attitude object or the self. These evaluations are generally either favorable or unfavorable. They come about from various influences in the individual experience. The commonly used definition of implicit attitude within cognitive and social psychology comes from Anthony Greenwald and Mahzarin Banaji's template for definitions of terms related to implicit cognition : "Implicit attitudes are introspectively unidentified traces of past experience that mediate favorable or unfavorable feeling, thought, or action toward social objects". These thoughts, feelings or actions have an influence on behavior that the individual may not be aware of.

Priming is a technique whereby exposure to one stimulus influences a response to a subsequent stimulus, without conscious guidance or intention. For example, the word NURSE is recognized more quickly following the word DOCTOR than following the word BREAD. Priming can be perceptual, semantic, or conceptual. Research, however, has yet to firmly establish the duration of priming effects, yet their onset can be almost instantaneous.

Mode effect is a broad term referring to a phenomenon where a particular survey administration mode causes different data to be collected. For example, when asking a question using two different modes, responses to one mode may be significantly and substantially different from responses given in the other mode. Mode effects are a methodological artifact, limiting the ability to compare results from different modes of collection.

Jon Krosnick American psychologist

Jon Alexander Krosnick is a professor of Political Science, Communication, and Psychology, and director of the Political Psychology Research Group (PPRG) at Stanford University. Additionally, he is the Frederic O. Glover Professor in Humanities and Social Sciences and an affiliate of the Woods Institute for the Environment. Krosnick has served as a consultant for government agencies, universities, and businesses, has testified as an expert in court proceedings, and has been an on-air television commentator on election night.

Political cognition refers to the study of how individuals come to understand the political world, and how this understanding leads to political behavior. Some of the processes studied under the umbrella of political cognition include attention, interpretation, judgment, and memory. Most of the advancements in the area have been made by scholars in the fields of social psychology, political science, and communication studies.

References

  1. 1 2 3 4 5 6 7 Zaller, J., & Feldman, S. (1992). A simple theory of survey response: Answering questions versus revealing preferences. American Journal of Political Science, 36(3), 579
  2. 1 2 Moy, P., Scheufele, D. A., Eveland, W. P. and McLeod, J. M. (2001), Support for the Death Penalty and Rehabilitation: Question Order or Communication Effect?1. Journal of Applied Social Psychology, 31: 2230–2255.
  3. Gladwell, M. (2001). The Art of Failure. The New Yorker, 84-92.
  4. Scheufele, D. A., & Tewksbury, D. (2007). Framing, agenda setting, and priming: The evolution of three media effects models. Journal of Communication, 57(1), 9-20.
  5. Ackerman, Joshua M., Christopher C. Nocera, and John A. Bargh. "Incidental haptic sensations influence social judgments and decisions." Science 328.5986 (2010): 1712-1715.
  6. Williams, Lawrence E., and John A. Bargh. "Experiencing physical warmth promotes interpersonal warmth." Science 322.5901 (2008): 606-607.
  7. Philip E. Converse (2006) The nature of belief systems in mass publics (1964), Critical Review, 18:1-3, 1-74.
  8. Achen, C. (1975). Mass Political Attitudes and the Survey Response. The American Political Science Review, 69(4), 1218-1231