Expert review (method)

Last updated

Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts.

An expert review has two primary goals: [1]

  1. Identify potential problems related to data quality and data collection so they can be mitigated.
  2. Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects.

An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions. [2] Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended. [1]

The experts document their assessment and typically report the results in open-ended comments. [3] They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error. [4] [5]

When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation. [6]

Compared to empirical pretesting methods, an expert review is more time-efficient and less costly to complete. [7] An expert review can also be combined with other methods to enable multimode pretesting. [8] [9] [10]

See also

Related Research Articles

<span class="mw-page-title-main">Research</span> Systematic study undertaken to increase knowledge

Research is "creative and systematic work undertaken to increase the stock of knowledge". It involves the collection, organization and analysis of evidence to increase understanding of a topic, characterized by a particular attentiveness to controlling sources of bias and error. These activities are characterized by accounting and controlling for biases. A research project may be an expansion of past work in the field. To test the validity of instruments, procedures, or experiments, research may replicate elements of prior projects or the project as a whole.

Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system. It is more concerned with the design intuitiveness of the product and tested with users who have no prior exposure to it. Such testing is paramount to the success of an end product as a fully functioning application that creates confusion amongst its users will not last for long. This is in contrast with usability inspection methods where experts use different methods to evaluate a user interface without involving users.

Marketing research is the systematic gathering, recording, and analysis of qualitative and quantitative data about issues relating to marketing products and services. The goal is to identify and assess how changing elements of the marketing mix impacts customer behavior.

<span class="mw-page-title-main">Interview</span> Structured series of questions and answers

An interview is a structured conversation where one participant asks questions, and the other provides answers. In common parlance, the word "interview" refers to a one-on-one conversation between an interviewer and an interviewee. The interviewer asks questions to which the interviewee responds, usually providing information. That information may be used or provided to other audiences immediately or later. This feature is common to many types of interviews – a job interview or interview with a witness to an event may have no other audience present at the time, but the answers will be later provided to others in the employment or investigative process. An interview may also transfer information in both directions.

Questionnaire construction refers to the design of a questionnaire to gather statistically useful information about a given topic. When properly constructed and responsibly administered, questionnaires can provide valuable data about any given subject.

Survey methodology is "the study of survey methods". As a field of applied statistics concentrating on human-research surveys, survey methodology studies the sampling of individual units from a population and associated techniques of survey data collection, such as questionnaire construction and methods for improving the number and accuracy of responses to surveys. Survey methodology targets instruments or procedures that ask one or more questions that may or may not be answered.

Qualitative marketing research involves a natural or observational examination of the philosophies that govern consumer behavior. The direction and framework of the research is often revised as new information is gained, allowing the researcher to evaluate issues and subjects in an in-depth manner. The quality of the research produced is heavily dependent on the skills of the researcher and is influenced by researcher bias.

The Delphi method or Delphi technique is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The technique can also be adapted for use in face-to-face meetings, and is then called mini-Delphi. Delphi has been widely used for business forecasting and has certain advantages over another structured forecasting approach, prediction markets.

<span class="mw-page-title-main">Questionnaire</span> Series of questions for gathering information

A questionnaire is a research instrument that consists of a set of questions for the purpose of gathering information from respondents through survey or statistical study. A research questionnaire is typically a mix of close-ended questions and open-ended questions. Open-ended, long-term questions offer the respondent the ability to elaborate on their thoughts. The Research questionnaire was developed by the Statistical Society of London in 1838.

<span class="mw-page-title-main">Response bias</span> Type of bias

Response bias is a general term for a wide range of tendencies for participants to respond inaccurately or falsely to questions. These biases are prevalent in research involving participant self-report, such as structured interviews or surveys. Response biases can have a large impact on the validity of questionnaires or surveys.

<span class="mw-page-title-main">Opinion</span> Judgment, viewpoint, or statement that is not conclusive

An opinion is a judgment, viewpoint, or statement that is not conclusive, rather than facts, which are true statements.

Cognitive tests are assessments of the cognitive capabilities of humans and other animals. Tests administered to humans include various forms of IQ tests; those administered to animals include the mirror test and the T maze test. Such testing is used in psychology and psychometrics, as well as other fields studying human and animal intelligence.

Cognitive pretesting, or cognitive interviewing, is a field research method where data is collected on how the subject answers interview questions. It is the evaluation of a test or questionnaire before it's administered. It allows survey researchers to collect feedback regarding survey responses and is used in evaluating whether the question is measuring the construct the researcher intends. The data collected is then used to adjust problematic questions in the questionnaire before fielding the survey to the full sample of people.

An expert report is a study written by one or more authorities that states findings and offers opinions.

With the application of probability sampling in the 1930s, surveys became a standard tool for empirical research in social sciences, marketing, and official statistics. The methods involved in survey data collection are any of a number of ways in which data can be collected for a statistical survey. These are methods that are used to collect information from a sample of individuals in a systematic way. First there was the change from traditional paper-and-pencil interviewing (PAPI) to computer-assisted interviewing (CAI). Now, face-to-face surveys (CAPI), telephone surveys (CATI), and mail surveys are increasingly replaced by web surveys. In addition, remote interviewers could possibly keep the respondent engaged while reducing cost as compared to in-person interviewers.

A translation project is a project that deals with the activity of translating.

The Quality of Well-Being Scale (QWB) is a general health quality of life questionnaire which measures overall status and well-being over the previous three days in four areas: physical activities, social activities, mobility, and symptom/problem complexes.

A vignette is a short description of one or more hypothetical characters or situation. They are used in quantitative surveys or in qualitative studies that pretest surveys.

Pretesting is testing and evaluating whether a questionnaire causes problems that could affect data quality and data collection for interviewers or survey respondents.

Behavior coding, or behavioral coding, is a research method for evaluating questionnaire design and survey interviewer performance and interaction.

References

  1. 1 2 Olson, Kristen (2010). "An Examination of Questionnaire Evaluation by Expert Reviewers". Field Methods . 22 (4): 295–318. doi:10.1177/1525822X10379795.
  2. DeMaio, Terry; Landreth, Ashley (2004), Prüfer, Peter; Rexroth, Margrit; Fowler, Floyd Jackson Jr. (eds.), "Examining expert reviews as a pretest method", QUEST 2003: proceedings of the 4th Conference on Questionnaire Evaluation Standards, 21–23 October 2003, ZUMA-Nachrichten Spezial, Mannheim: Zentrum für Umfragen, Methoden und Analysen -ZUMA-, vol. 9, pp. 60–73, ISBN   978-3-924220-27-3 , retrieved 2024-02-21
  3. Yan, T.; Kreuter, F.; Tourangeau, R (December 2012). "Evaluating Survey Questions: A Comparison of Methods". Journal of Official Statistics . 28 (4): 503–529.
  4. Willis, G; Lessler, J (1999). "Question Appraisal System (QAS-99)" (PDF). Centers for Disease Control and Prevention Website.
  5. Maitland, Aaron; Presser, Stanley (2020-01-02), Beatty, Paul; Collins, Debbie; Kaye, Lyn; Padilla, Jose Luis (eds.), "A Comparison of Five Question Evaluation Methods in Predicting the Validity of Respondent Answers to Factual Items", Advances in Questionnaire Design, Development, Evaluation and Testing (1 ed.), Wiley, pp. 75–90, doi:10.1002/9781119263685.ch4, ISBN   978-1-119-26362-3 , retrieved 2024-02-21
  6. Pan, Yuling; Sha, Mandy (2019-07-09). The Sociolinguistics of Survey Translation. London: Routledge. pp. 118–124. doi:10.4324/9780429294914. ISBN   978-0-429-29491-4. S2CID   198632812.
  7. Presser, S; Blair, J. "Survey pretesting: Do different methods produce different results?". Sociological Methodology . 24: 73–104.
  8. Ridolfo, Heather; Thompson, Ashley (2023-10-05). "Conducting Small-Scale Multimethod Questionnaire Evaluation". Survey Practice. 16 (1). doi: 10.29115/SP-2023-0017 .
  9. Sha, Mandy (March 2012). "Translation Review and Cognitive Testing of ACS Language Assistance Guides in Multiple Languages". United States Census Bureau. Retrieved 2024-01-08.
  10. Aizpurua, Eva (2020). "Pretesting methods in cross-cultural research (Chapter 7)". In Sha, Mandy; Gabel, Tim (eds.). The essential role of language in survey research. RTI Press. pp. 129–150. doi: 10.3768/rtipress.bk.0023.2004 . ISBN   978-1-934831-23-6.