Designing Social Inquiry

Last updated March 21, 2023

Designing Social Inquiry: Scientific Inference in Qualitative Research (or KKV) is an influential 1994 book written by Gary King, Robert Keohane, and Sidney Verba that lays out guidelines for conducting qualitative research.^[1] The central thesis of the book is that qualitative and quantitative research share the same "logic of inference."^[2] The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research.^[3]^[1]

History

Robert Keohane recounts the origins of KKV as follows,^[7]

Designing Social Inquiry was not generated by puzzles of world politics. Instead, it was the result of serendipity. Sid Verba and I were friends, and when I joined the Harvard Government Department in 1985, he said that we should teach a course together. I regarded this remark as a welcoming pleasantry, typical of Sid's grace and warmth. Three years later I became chair of the department and in my first year as chair was forced to listen to 24 job talks. Most of these talks were dead on arrival, since the speaker had made fundamental mistakes in research design. I complained to colleagues, including Sid, and Gary King. Gary said the three of us should teach a course on research design together... I agreed, and we taught the course the following year... After the semester was over, Gary said: “We should teach the course again. And this time, we should write a book on this subject.” The next year we met regularly for a bag lunch, discussing not only themes of the course but drafts that one of us—most often Gary, which is why his name appears first on the book—had produced.

The goal of the book is guide researchers in producing valid causal inferences in social science research.^[8] The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research.^[3]^[1] The authors argue “whether we study many phenomena or few… the study will be improved if we collect data on as many observable implications of our theory as possible.”^[8] The authors note that case studies do not necessarily have to be N=1 or few N: a case study can include many observations within a case (many individuals and entities across many time periods).^[9] KKV criticize Harry H. Eckstein's notion of "crucial case studies", warning that a single observation makes it harder to estimate multiple causal effects, more likely that there is measurement error, and risks that an event in a single case was caused by random error.^[10]

According to the authors, a strong research design requires both qualitative and quantitative research, a research question that poses an important and real question that will contribute to the base of knowledge about this particular subject, and a comprehensive literature review from which hypotheses (theory-driven) are then drawn. Data that are collected should be operationalized so that other researchers could replicate the study and achieve similar results. For the same reason, the reasoning process behind the analysis needs to be explicit. While gathering data the researcher should consider the observable implications of the theory in an effort to explain as much of the data as possible. This is in addition to examining the causal mechanisms that connect one variable to another.

While qualitative methods cannot produce precise measurements of uncertainty about the conclusions (unlike quantitative methods), qualitative scholars should give indications about the uncertainty of their inferences. KKV argue that "the single most serious problem with qualitative research in political science is the pervasive failure to provide reasonable estimates of the uncertainty of the investigator’s inferences."^[11]

According to KKV, the rules for good causal theories are that they need to:

be falsifiable
have internal consistency (generate hypotheses that do not contradict each other)
have variation (explanatory variables should be exogenous and dependent variables should be endogenous)
have "concrete" concepts (concepts should be observable)
have "leverage" (the theory should explain much by little).^[12]

KKV sees process-tracing and qualitative research as being "unable to yield strong causal inference" due to the fact that qualitative scholars would struggle with determining which of many intervening variables truly links the independent variable with a dependent variable. The primary problem is that qualitative research lacks a sufficient number of observations to properly estimate the effects of an independent variable. They write that the number of observations could be increased through various means, but that would simultaneously lead to another problem: that the number of variables would increase and thus reduce degrees of freedom.^[1]

In terms of case selection, KKV warn against "selecting on the dependent variable". For example, researchers cannot make valid causal inferences about wars outbreak by only looking at instances where war did happen (the researcher should also look at cases where war did not happen). There is methodological problem in selecting on the explanatory variable, however. They do warn about multicollinearity (choosing two or more explanatory variables that perfectly correlate with each other). They argue that random selection of cases is a valid case selection strategy in large-N research, but warn against it in small-N research.

KKV reject the notion of "quasi-experiments", arguing that either all the key causal variables can be controlled (an experiment) or not (a non-experiment).

Reception

In his 2010 review, James Mahoney writes that the field of social science methodology has "benefited from KKV even as it has also moved beyond it."^[1] Critics of KKV have characterized the book's claims as "often simplistic, misleading and inappropriate as a guide for designing social inquiry."^[1] Quantitative scholars such as Henry E. Brady, Larry M. Bartels and David A. Freedman have argued that KKV overstate the strengths of quantitative research vis-a-vis qualitative research.^[1] Henry Brady and David Collier argue that KKV exaggerate the ability of quantitative research to identify uncertainty.^[5] They also argue that KKV exaggerate the risks of conducting inductive research and forming hypotheses post hoc.^[5]

Numerous scholars disagree with KKV in their claims that qualitative research should integrate standards from quantitative research.^[5]^[6]^[13] There are different logics to the manner in which qualitative research is conducted and what qualitative scholars seek and can do with their data.^[13] Brady and Collier argue that KKV give insufficient attention to these divergent logics, as well as the intrinsic tradeoffs between different methodological goals.^[5] Gary Goertz and James Mahoney dispute that the main difference between qualitative and quantitative research is the size of N. Instead, a primary difference is that qualitative scholars tend to do within-case analyses whereas quantitative scholars almost by definition do cross-case analyses.^[13]

Mahoney writes that KKV ignore set theory and logic in terms of evaluating causal inference. Whereas regression-oriented analyses seek to estimate average effects of certain outcomes, qualitative research seeks to explain why cases have certain outcomes.^[1] Thus, causal inference is not strengthened by expanding the size of N, but rather by carefully choosing cases, whose testing can strengthen or weaken a theory.^[1] Mahoney and Gary Goertz make an analogy with a murder case: a single piece of smoking gun evidence can conclusively show whether a person committed a murder.^[13]

Mahoney also writes that KKV give insufficient attention to concept formation, which is an essential aspect of theory construction and measurement, and one of the important ways that qualitative research can play a key role.^[1]

Ronald Rogowski criticizes how KKV treat qualitative social science research. Rogowski argues that there is too much focus on hypothesis-testing and too much caution against using single observations. Rogowski argues that KKV promotes a form of qualitative social science that is overly focused on hypothesis-testing, and that this limits scholars' questions, cases and ambitions.^[14]^[15] John J. Mearsheimer and Stephen M. Walt argue that International Relations scholarship has shifted away from crafting and refining IR theory to "simplistic hypothesis-testing", in part due to the influence of KKV in political science graduate programs.^[16]

Alexander George and Andrew Bennett say there is "much to agree with" in KKV, but they argue that the book has several flaws in its guidance on qualitative research:^[6]

Causal mechanisms: KKV suggest that "causal mechanisms" are less important that "causal effects" in causal explanations – George and Bennett argue that they are equally important
Hypothesis-testing: KKV overly emphasize the role of hypothesis-testing in theory development – George and Bennett argue that the formation of new hypotheses and the choosing of new questions are also important aspects of theory development
Causal complexity: KKV fail to consider problems of causal complexity, such as equifinality, multiple interaction effects, feedback loops, path dependency, tipping points, selection effects, expectations effects and sequential interactions – George and Bennet argue that case studies, process tracing and typological theories can clarify causality in situations of causal complexity
Increasing N: KKV argue that scholars should always seek to increase the number of cases – George and Bennett argue that KKV fail to consider the costs to increasing the number of cases (such as conceptual stretching and unintentional comparisons of dissimilar cases). George and Bennett note that much value can be derived from single-case studies.
Process-tracing: KKV characterize process-tracing as a way to increase the number of observable implications – George and Bennett argue that the logic of process-tracing is entirely different. The logic behind using process-tracing is to focus on the sequences and timings within a case, not to correlate data across cases. Thus, if one piece of evidence in the sequence is inconsistent with the theoretical expectations, then the theory has been shown to be flawed.
"Degrees of freedom" problem: KKV argue that a single case cannot evaluate competing explanations due problems that arise from degrees of freedom – George and Bennett argue that it is flawed to apply this statistical logic to qualitative research. George and Bennett say that while quantitative scholars try to aggregate variables to reduce the number of variables and thus increase the degrees of freedom, qualitative scholars intentionally want their variables to have many different attributes and complexity.

Related Research Articles

A case study is an in-depth, detailed examination of a particular case within a real-world context. For example, case studies in medicine may focus on an individual patient or ailment; case studies in business might cover a particular firm's strategy or a broader market; similarly, case studies in politics can range from a narrow happening over time to an enormous undertaking.

Multimethodology or multimethod research includes the use of more than one method of data collection or research in a research study or set of related studies. Mixed methods research is more specific in that it includes the mixing of qualitative and quantitative data, methods, methodologies, and/or paradigms in a research study or set of related studies. One could argue that mixed methods research is a special case of multimethod research. Another applicable, but less often used label, for multi or mixed research is methodological pluralism. All of these approaches to professional and academic research emphasize that monomethod research can be improved through the use of multiple data sources, methods, research methodologies, perspectives, standpoints, and paradigms.

Qualitative research is a type of research that aims to gather and analyse non-numerical (descriptive) data in order to gain an understanding of individuals' social reality, including understanding their attitudes, beliefs, and motivation. This type of research typically involves in-depth interviews, focus groups, or observations in order to collect data that is rich in detail and context. Qualitative research is often used to explore complex phenomena or to gain insight into people's experiences and perspectives on a particular topic. It is particularly useful when researchers want to understand the meaning that people attach to their experiences or when they want to uncover the underlying reasons for people's behavior. Qualitative methods include ethnography, grounded theory, discourse analysis, and interpretative phenomenological analysis. Qualitative research methods have been used in sociology, anthropology, political science, psychology, social work, folklore, educational research and software engineering research.

Social research is a research conducted by social scientists following a systematic plan. Social research methodologies can be classified as quantitative and qualitative.

Quantitative research is a research strategy that focuses on quantifying the collection and analysis of data. It is formed from a deductive approach where emphasis is placed on the testing of theory, shaped by empiricist and positivist philosophies.

Content analysis is the study of documents and communication artifacts, which might be texts of various formats, pictures, audio or video. Social scientists use content analysis to examine patterns in communication in a replicable and systematic manner. One of the key advantages of using content analysis to analyse social phenomena is its non-invasive nature, in contrast to simulating social experiences or collecting survey answers.

Comparative politics is a field in political science characterized either by the use of the comparative method or other empirical methods to explore politics both within and between countries. Substantively, this can include questions relating to political institutions, political behavior, conflict, and the causes and consequences of economic development. When applied to specific fields of study, comparative politics may be referred to by other names, such as comparative government.

Exploratory research is "the preliminary research to clarify the exact nature of the problem to be solved." It is used to ensure additional research is taken into consideration during an experiment as well as determining research priorities, collecting data and honing in on certain subjects which may be difficult to take note of without exploratory research. It can include techniques, such as:

<i>States and Social Revolutions</i> 1979 book by Theda Skocpol

States and Social Revolutions: A Comparative Analysis of France, Russia and China is a 1979 book by Theda Skocpol, published by Cambridge University Press, that examines the causes of social revolutions.

Robert Owen Keohane is an American academic working within the fields of international relations and international political economy. Following the publication of his influential book After Hegemony (1984), he has become widely associated with the theory of neoliberal institutionalism in international relations, as well as transnational relations and world politics in international relations in the 1970s.

Helen V. Milner is an American political scientist and the B. C. Forbes Professor of Public Affairs at the Woodrow Wilson School of Public and International Affairs at Princeton University, where she is also the Director of the Niehaus Center for Globalization and Governance. She has written extensively on issues related to international political economy like international trade, the connections between domestic politics and foreign policy, globalization and regionalism, and the relationship between democracy and trade policy.

Gary King is an American political scientist and quantitative methodologist. He is the Albert J. Weatherhead III University Professor and Director for the Institute for Quantitative Social Science at Harvard University. King and his research group develop and apply empirical methods in many areas of social science research, focusing on innovations that span the range from statistical theory to practical application.

Sidney Verba was an American political scientist, librarian and library administrator. His academic interests were mainly American and comparative politics. He was the Carl H. Pforzheimer University Professor at Harvard University and also served Harvard as the director of the Harvard University Library from 1984 to 2007.

Comparative historical research is a method of social science that examines historical events in order to create explanations that are valid beyond a particular time and place, either by direct comparison to other historical events, theory building, or reference to the present day. Generally, it involves comparisons of social processes across times and places. It overlaps with historical sociology. While the disciplines of history and sociology have always been connected, they have connected in different ways at different times. This form of research may use any of several theoretical orientations. It is distinguished by the types of questions it asks, not the theoretical framework it employs.

In statistics, qualitative comparative analysis (QCA) is a data analysis based on set theory to examine the relationship of conditions to outcome. QCA describes the relationship in terms of necessary conditions and sufficient conditions. The technique was originally developed by Charles Ragin in 1987 to study data sets that are too small for linear regression analysis but large for cross-case analysis.

David Collier is an American political scientist specializing in comparative politics. He is Chancellor's Professor Emeritus at the University of California, Berkeley. He works in the fields of comparative politics, Latin American politics, and methodology. His father was the anthropologist Donald Collier.

Process tracing is a research method used to develop and test theories. It is generally understood as a "within-case" method to draw inferences on the basis of causal mechanisms. It has been used in social sciences, as well as in natural sciences.

Causal inference is the process of determining the independent, actual effect of a particular phenomenon that is a component of a larger system. The main difference between causal inference and inference of association is that causal inference analyzes the response of an effect variable when a cause of the effect variable is changed. The science of why things occur is called etiology. Causal inference is said to provide the evidence of causality theorized by causal reasoning.

Feminist empiricism is a perspective within feminist research that combines the objectives and observations of feminism with the research methods and empiricism. Feminist empiricism is typically connected to mainstream notions of positivism. Feminist empiricism critiques what it perceives to be inadequacies and biases within mainstream research methods, including positivism.

Harry H. Eckstein was an American political scientist. He was an influential scholar of comparative politics and political culture, as well as qualitative research methods.

References

1 2 3 4 5 6 7 8 9 10 Mahoney, James (2010). "After KKV: The New Methodology of Qualitative Research". World Politics. 62 (1): 120–147. doi:10.1017/S0043887109990220. ISSN 1086-3338. S2CID 43923978.
↑ King, Gary/ Keohane, Robert O./ Verba, Sidney: Designing Social Inquiry. Scientific Inference in Qualitative Research, p. 3. Princeton University Press, 1994.
1 2 Humphreys, Macartan; Jacobs, Alan M. (2015). "Mixing Methods: A Bayesian Approach". American Political Science Review. 109 (4): 654. doi:10.1017/s0003055415000453. ISSN 0003-0554. S2CID 1846974.
↑ Morgan, Kimberly J. (2016-09-02). "Process tracing and the causal identification revolution". New Political Economy. 21 (5): 489–492. doi:10.1080/13563467.2016.1201804. ISSN 1356-3467. S2CID 156377366.
1 2 3 4 5 6 Brady, Henry E. Collier, David (2010). Rethinking social inquiry : diverse tools, shared standards (2 ed.). Rowman & Littlefield Publishers. ISBN 978-1-4422-0343-3. OCLC 838295613.{{cite book}}: CS1 maint: multiple names: authors list (link)
1 2 3 George, Alexander L.; Bennett, Andrew (2005). Case Studies and Theory Development in the Social Sciences. MIT Press. pp. 10–18, 28–33. ISBN 978-0-262-30307-1. OCLC 944521872.
↑ Keohane, Robert O. (2020-05-11). "Understanding Multilateral Institutions in Easy and Hard Times". Annual Review of Political Science. 23 (1): 1–18. doi: 10.1146/annurev-polisci-050918-042625 . ISSN 1094-2939.
1 2 King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 1–4, 12. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
↑ King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 52–53. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
↑ King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 209–211. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
↑ King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. p. 32. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
↑ King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 99–114. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.
1 2 3 4 Goertz, Gary; Mahoney, James (2012-09-09). A Tale of Two Cultures. Princeton University Press. pp. 1–2, 10. doi:10.23943/princeton/9780691149707.001.0001. ISBN 978-0-691-14970-7.
↑ Rogowski, Ronald (2010). "How Inference in the Social (but Not the Physical) Sciences Neglects Theoretical Anomaly" in Rethinking social inquiry diverse tools, shared standards. Rowman & Littlefield Publishers. ISBN 978-1-4422-0343-3. OCLC 787870333.
↑ Rogowski, Ronald (1995). "The Role of Theory and Anomaly in Social-Scientific Inference". American Political Science Review. 89 (2): 467–470. doi:10.2307/2082443. ISSN 1537-5943. JSTOR 2082443. S2CID 146265258.
↑ Mearsheimer, John J.; Walt, Stephen M. (2013). "Leaving theory behind: Why simplistic hypothesis testing is bad for International Relations". European Journal of International Relations. 19 (3): 427–457. doi:10.1177/1354066113494320. ISSN 1354-0661. S2CID 52247884.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[:0-1] 1 2 3 4 5 6 7 8 9 10 Mahoney, James (2010). "After KKV: The New Methodology of Qualitative Research". World Politics. 62 (1): 120–147. doi:10.1017/S0043887109990220. ISSN 1086-3338. S2CID 43923978.

[2] King, Gary/ Keohane, Robert O./ Verba, Sidney: Designing Social Inquiry. Scientific Inference in Qualitative Research, p. 3. Princeton University Press, 1994.

[:1-3] 1 2 Humphreys, Macartan; Jacobs, Alan M. (2015). "Mixing Methods: A Bayesian Approach". American Political Science Review. 109 (4): 654. doi:10.1017/s0003055415000453. ISSN 0003-0554. S2CID 1846974.

[4] Morgan, Kimberly J. (2016-09-02). "Process tracing and the causal identification revolution". New Political Economy. 21 (5): 489–492. doi:10.1080/13563467.2016.1201804. ISSN 1356-3467. S2CID 156377366.

[:3-5] 1 2 3 4 5 6 Brady, Henry E. Collier, David (2010). Rethinking social inquiry : diverse tools, shared standards (2 ed.). Rowman & Littlefield Publishers. ISBN 978-1-4422-0343-3. OCLC 838295613.{{cite book}}: CS1 maint: multiple names: authors list (link)

[:4-6] 1 2 3 George, Alexander L.; Bennett, Andrew (2005). Case Studies and Theory Development in the Social Sciences. MIT Press. pp. 10–18, 28–33. ISBN 978-0-262-30307-1. OCLC 944521872.

[7] Keohane, Robert O. (2020-05-11). "Understanding Multilateral Institutions in Easy and Hard Times". Annual Review of Political Science. 23 (1): 1–18. doi: 10.1146/annurev-polisci-050918-042625 . ISSN 1094-2939.

[:2-8] 1 2 King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 1–4, 12. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.

[9] King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 52–53. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.

[10] King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 209–211. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.

[11] King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. p. 32. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.

[12] King, Gary; Keohane, Robert O.; Verba, Sidney (1994). Designing Social Inquiry. Princeton, New Jersey: Princeton University Press. pp. 99–114. doi:10.1515/9781400821211. ISBN 978-1-4008-2121-1.

[:5-13] 1 2 3 4 Goertz, Gary; Mahoney, James (2012-09-09). A Tale of Two Cultures. Princeton University Press. pp. 1–2, 10. doi:10.23943/princeton/9780691149707.001.0001. ISBN 978-0-691-14970-7.

[14] Rogowski, Ronald (2010). "How Inference in the Social (but Not the Physical) Sciences Neglects Theoretical Anomaly" in Rethinking social inquiry diverse tools, shared standards. Rowman & Littlefield Publishers. ISBN 978-1-4422-0343-3. OCLC 787870333.

[15] Rogowski, Ronald (1995). "The Role of Theory and Anomaly in Social-Scientific Inference". American Political Science Review. 89 (2): 467–470. doi:10.2307/2082443. ISSN 1537-5943. JSTOR 2082443. S2CID 146265258.

[16] Mearsheimer, John J.; Walt, Stephen M. (2013). "Leaving theory behind: Why simplistic hypothesis testing is bad for International Relations". European Journal of International Relations. 19 (3): 427–457. doi:10.1177/1354066113494320. ISSN 1354-0661. S2CID 52247884.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]