Wiki survey

Last updated
A view of the 'opinion space' visualization tool for a Polis wiki survey. Each user can see how similar their voting behavior is to other voters based on their closeness to one another within the two-dimensional space. Polis Wiki Survey Opinion Space Reduced Resolution.png
A view of the 'opinion space' visualization tool for a Polis wiki survey. Each user can see how similar their voting behavior is to other voters based on their closeness to one another within the two-dimensional space.

Wiki surveys or wikisurveys are a software-based survey method with similarity to how wikis evolve through crowdsourcing. In essence, they are surveys that allow participants to create the questions that are being asked. [1] [2] [3] As participants engage in the survey they can either vote on a survey question or create a survey question. A single open-ended prompt written by the creator of the survey determines the topic the questions should be on. The first known implementation of a wiki survey was in 2010, [4] and they have been used since then for a variety of purposes such as facilitating deliberative democracy, crowdsourcing opinions from experts and figuring out common beliefs on a given topic. [5] [6] [7] A notable usage of wiki surveys is in Taiwan's government system, where citizens can participate in crowdsourced lawmaking through Polis wiki surveys. [8] [9] [10]

Contents

Wiki surveys facilitate collective intelligence by allowing users to both contribute and respond to the survey, as well as see the results of the survey in real time. They can be seen in a more general sense as a tool for establishing consensus in large volumes of people. Wiki surveys mainly differ from consensus-building in comment sections by using a heuristic which determines the order of questions for each participant that aims to maximize consensus, not allowing replies to questions and providing visualization tools to better understand consensus.

Implementations

All Our Ideas

A view of the user interface for seeing in real-time the results of an All Our Ideas wiki survey. The top 10 ranked items are shown in this view of the interface. All our ideas wiki pic 2 reduced resolution.png
A view of the user interface for seeing in real-time the results of an All Our Ideas wiki survey. The top 10 ranked items are shown in this view of the interface.
A view of the user interface for casting votes on and submitting items to an All Our Ideas wiki survey. After casting a vote, a heuristic determines the next item that will be shown. The heuristic favors showing items with relatively low numbers of votes. All our ideas wiki pic 1 reduced resolution.png
A view of the user interface for casting votes on and submitting items to an All Our Ideas wiki survey. After casting a vote, a heuristic determines the next item that will be shown. The heuristic favors showing items with relatively low numbers of votes.

All Our Ideas was the first ever wiki survey. [1] Its focus is on ranking the favorability of each 'item' that users submit to the survey. Each question presented asks the participant to rank the best of two items. At any point in time, participants can view a ranking of the items in order of their score. The score for an item is the estimated probability that it would be favored over another randomly chosen item. In this sense, it is considered a 'pairwise wiki survey'. The code for All Our Ideas is open source. [11]

Polis

Polis (also known as Pol.is) was developed in 2012. [2] The focus of Polis is to project participants into an 'opinion space' where they can see how their voting behavior compares to other participants. The opinion space clusters participants into groups of similar opinion and is designed in a way to avoid tyranny of the majority by being able to include groups that have small numbers of participants. The questions participants are presented with are a simple agree/disagree/pass on a single 'comment' submitted by a participant. The code for Polis is open source. [12]

Examples

Characteristics

Wiki surveys have the following three defining characteristics: [1] [5] [2]

Collaborativeness

Wiki surveys allow participants to contribute questions, as well as answer questions created by its participants.

Adaptivity

Wiki surveys adapt to elicit the most useful information from its participants. This is done by changing the ordering of questions based on the voting behavior of previous participants so as to maximize consensus. The heuristic determining the ordering of questions highly values showing the comments that have been voted on the least.

Greediness

Although being greedy typically has a negative connotation, it is used in a positive manner for wiki surveys. They are 'greedy' because they make full use of information that participants are willing to provide. Wiki surveys do not require participants to answer a fixed amount of questions, participants can answer as little or as much as they want.

Traditional survey methods vs. wiki surveys

Advantages of wiki surveys

Questions in traditional survey methods fall into two categories: Open and closed questions. Open questions ask the person taking the survey to write an open response while closed questions give a fixed set of responses to select from. [18] Wiki surveys are like a hybrid of the two, enabling insightful consensus in certain situations where traditional survey methods may lack. Closed questions are easy to analyze quantitively, but the limited options to select from for a given question may cause bias. Open questions are not as subject to bias, but are difficult to analyze quantitatively at scale. Wiki surveys allow for open responses by the users' contribution of survey questions (also called 'items'), and uses machine learning techniques to automatically quantitative analyze the responses to those questions.

The 'greediness' characteristic of wiki surveys is thought to also be advantageous, as it allows for gathering more data per participant. [1] Data from real-world usage of wiki surveys shows that there are a relatively small amount of participants that answer a relatively large amount of questions compared to most participants. In traditional surveys, participants who want to provide more information than required are typically not allowed to do so, thus missing out on potentially useful information.

Disadvantages of wiki surveys

Traditional survey methods are better suited for situations where the survey creator(s) need to acquire consensus on a specific question or set of questions. Wiki surveys are more seen as a method for deliberation and gathering consensus on ideas that were not thought of by the survey creator(s). There is also a lack of research on determining potential biases that wiki surveys may cause.[ citation needed ]

Related Research Articles

<span class="mw-page-title-main">Consensus decision-making</span> Group decision-making aiming for universal agreement

Consensus decision-making or consensus process are group decision-making processes in which participants develop and decide on proposals with the goal of acceptance by all. The focus on establishing agreement of at least the majority or the supermajority and avoiding unproductive opinion differentiates consensus from unanimity, which requires all participants to support a decision. Consensus decision-making in a democracy is consensus democracy.

Deliberative democracy or discursive democracy is a form of democracy in which deliberation is central to decision-making. Deliberative democracy seeks quality over quantity by limiting decision-makers to a smaller but more representative sample of the population that is given the time and resources to focus on one issue.

Survey methodology is "the study of survey methods". As a field of applied statistics concentrating on human-research surveys, survey methodology studies the sampling of individual units from a population and associated techniques of survey data collection, such as questionnaire construction and methods for improving the number and accuracy of responses to surveys. Survey methodology targets instruments or procedures that ask one or more questions that may or may not be answered.

An opinion poll, often simply referred to as a survey or a poll, is a human research survey of public opinion from a particular sample. Opinion polls are usually designed to represent the opinions of a population by conducting a series of questions and then extrapolating generalities in ratio or within confidence intervals. A person who conducts polls is referred to as a pollster.

The Delphi method or Delphi technique is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The technique can also be adapted for use in face-to-face meetings, and is then called mini-Delphi. Delphi has been widely used for business forecasting and has certain advantages over another structured forecasting approach, prediction markets.

<span class="mw-page-title-main">Questionnaire</span> Series of questions for gathering information

A questionnaire is a research instrument that consists of a set of questions for the purpose of gathering information from respondents through survey or statistical study. A research questionnaire is typically a mix of close-ended questions and open-ended questions. Open-ended, long-term questions offer the respondent the ability to elaborate on their thoughts. The Research questionnaire was developed by the Statistical Society of London in 1838.

<i>The Wisdom of Crowds</i> 2004 book by James Surowiecki

The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations, published in 2004, is a book written by James Surowiecki about the aggregation of information in groups, resulting in decisions that, he argues, are often better than could have been made by any single member of the group. The book presents numerous case studies and anecdotes to illustrate its argument, and touches on several fields, primarily economics and psychology.

Open-source governance is a political philosophy which advocates the application of the philosophies of the open-source and open-content movements to democratic principles to enable any interested citizen to add to the creation of policy, as with a wiki document. Legislation is democratically opened to the general citizenry, employing their collective wisdom to benefit the decision-making process and improve democracy.

Amazon Mechanical Turk (MTurk) is a crowdsourcing website with which businesses can hire remotely located "crowdworkers" to perform discrete on-demand tasks that computers are currently unable to do as economically. It is operated under Amazon Web Services, and is owned by Amazon. Employers post jobs known as Human Intelligence Tasks (HITs), such as identifying specific content in an image or video, writing product descriptions, or answering survey questions. Workers, colloquially known as Turkers or crowdworkers, browse among existing jobs and complete them in exchange for a fee set by the employer. To place jobs, the requesting programs use an open application programming interface (API), or the more limited MTurk Requester site. As of April 2019, Requesters could register from 49 approved countries.

Electronic participation (e-participation) refers to the use of ICT in facilitating citizen participation in government-related processes, encompassing areas such as administration, service delivery, decision-making, and policy-making. As such, e-participation shares close ties with e-government and e-governance participation. The term's emergence aligns with the digitization of citizen interests and interactions with political service providers, primarily due to the proliferation of e-government.

<span class="mw-page-title-main">Crowdsourcing</span> Sourcing services or funds from a group

Crowdsourcing involves a large group of dispersed participants contributing or producing goods or services—including ideas, votes, micro-tasks, and finances—for payment or as volunteers. Contemporary crowdsourcing often involves digital platforms to attract and divide work between participants to achieve a cumulative result. Crowdsourcing is not limited to online activity, however, and there are various historical examples of crowdsourcing. The word crowdsourcing is a portmanteau of "crowd" and "outsourcing". In contrast to outsourcing, crowdsourcing usually involves less specific and more public groups of participants.

Computer-assisted web interviewing (CAWI) is an Internet surveying technique in which the interviewee follows a script provided in a website. The questionnaires are made in a program for creating web interviews. The program allows for the questionnaire to contain pictures, audio and video clips, links to different web pages, etc. The website is able to customize the flow of the questionnaire based on the answers provided, as well as information already known about the participant. It is considered to be a cheaper way of surveying since one doesn't need to use people to hold surveys unlike computer-assisted telephone interviewing. With the increasing use of the Internet, online questionnaires have become a popular way of collecting information. The design of an online questionnaire has a dramatic effect on the quality of data gathered. There are many factors in designing an online questionnaire; guidelines, available question formats, administration, quality and ethic issues should be reviewed. Online questionnaires should be seen as a sub-set of a wider-range of online research methods.

Google Moderator was a Google service that used crowdsourcing to rank user-submitted questions, suggestions and ideas. It was launched on September 25, 2008 and shut down on June 30, 2015. The service allowed the management of feedback from a large number of people, who could vote for questions they thought should be posed from a pool of questions submitted by others or submit their own to be asked and voted on. The service aimed to ensure that every question was considered, let the audience see others' questions, and helped the moderator of a team or event address the questions that the audience most cared about. The service was nicknamed Dory internally by Google, a reference to "the fish who asked questions all the time in Finding Nemo".

Citizen sourcing is the government adoption of crowdsourcing techniques for the purposes of (1) enlisting citizens in the design and execution of government services and (2) tapping into the citizenry's collective intelligence for solutions and situational awareness. Applications of citizen sourcing include:

Crowdsourcing software development or software crowdsourcing is an emerging area of software engineering. It is an open call for participation in any task of software development, including documentation, design, coding and testing. These tasks are normally conducted by either members of a software enterprise or people contracted by the enterprise. But in software crowdsourcing, all the tasks can be assigned to or are addressed by members of the general public. Individuals and teams may also participate in crowdsourcing contests.

Government crowdsourcing is a form of crowdsourcing employed by governments to better leverage their constituents' collective knowledge and experience. It has tended to take the form of public feedback, project development, or petitions in the past, but has grown to include public drafting of bills and constitutions, among other things. This form of public involvement in the governing process differs from older systems of popular action, from town halls to referendums, in that it is primarily conducted online or through a similar IT medium.

Q&A software is online software that attempts to answer questions asked by users. Q&A software is frequently integrated by large and specialist corporations and tends to be implemented as a community that allows users in similar fields to discuss questions and provide answers to common and specialist questions.

Crowdsource is a crowdsourcing platform developed by Google intended to improve a host of Google services through the user-facing training of different algorithms.

Crowdsourced science refers to collaborative contributions of a large group of people to the different steps of the research process in science. In psychology, the nature and scope of the collaborations can vary in their application and in the benefits it offers.

References

  1. 1 2 3 4 5 6 Salganik, Matthew J.; Levy, Karen E. C. (2015-05-20). Helleringer, Stephane (ed.). "Wiki Surveys: Open and Quantifiable Social Data Collection". PLOS ONE. 10 (5): e0123483. Bibcode: 2015PLoSO..1023483S . doi: 10.1371/journal.pone.0123483 . ISSN   1932-6203. PMC   4439069 . PMID   25992565.
  2. 1 2 3 4 Small, Christopher; Bjorkegren, Michael; Erkkilä, Timo; Shaw, Lynette; Megill, Colin (2021-07-22). "Polis: Scaling deliberation by mapping high dimensional opinion spaces". Democracy in the Age of Artificial Intelligence. 26 (2). doi: 10.6035/recerca.5516 . hdl: 10234/196348 . ISSN   2254-4135. S2CID   242371333 .
  3. Bass, Theo (28 January 2019). "Crowdsourcing for democracy using Wikisurveys". nesta. Retrieved 2022-02-28.
  4. "About this project". All Our Ideas. Retrieved 2022-02-28.
  5. 1 2 French, Megan; Hancock, Jeff (2017-02-02). "What's the Folk Theory? Reasoning About Cyber-Social Systems". SSRN. Rochester, NY. SSRN   2910571.
  6. Wong, Arnold YL; Lauridsen, Henrik H.; Samartzis, Dino; Macedo, Luciana; Ferreira, Paulo H.; Ferreira, Manuela L. (2019-01-15). "Global Consensus From Clinicians Regarding Low Back Pain Outcome Indicators for Older Adults: Pairwise Wiki Survey Using Crowdsourcing". JMIR Rehabilitation and Assistive Technologies. 6 (1): e11127. doi: 10.2196/11127 . PMC   6350088 . PMID   30664493. Archived from the original on Jun 16, 2022.
  7. "Featured Case Studies". The Computational Democracy Project. Retrieved 2022-02-28.
  8. Horton, Chris (August 21, 2018). "The simple but ingenious system Taiwan uses to crowdsource its laws". MIT Technology Review. Retrieved 2022-02-28.
  9. BBC Click (Oct 29, 2019), "Can Taiwan Reboot Democracy?", YouTube, retrieved 2022-02-28
  10. Harris, Tristan; Raskin, Aza; Tang, Audrey (July 23, 2020). "22 – Digital Democracy is Within Reach with Audrey Tang". Your Undivided Attention. Center for Humane Technology. Retrieved 2022-03-18.
  11. "All Our Ideas". GitHub. Retrieved 2022-03-20.
  12. Polis, The Computational Democracy Project, 2022-03-15, retrieved 2022-03-20
  13. Ruiz, Rebecca (2018-01-20). "Nasty women (and men) to crowdsource 'marching orders' ahead of midterm elections". Mashable. Retrieved 2022-03-20.
  14. "AARP and All Our Ideas". congress.crowd.law. Retrieved 2022-03-20.
  15. "Testimony of Beth Simone Noveck, Professor and Director, The Governance Lab, New York University U.S. House Select Committee on the Modernization of Congress Recommendations for the Committee's agenda for the 117th Congress" (PDF). US House of Representatives. March 25, 2021.
  16. "Civic Assembly". Bowling Green Daily News. Retrieved 2022-03-21.
  17. "Anonymous voting tool aims to find Harrogate's consensus on key issues". York Press. Retrieved 2022-03-25.
  18. Schuman, Howard; Scott, Jacqueline (1987-05-22). "Problems in the Use of Survey Questions to Measure Public Opinion". Science. 236 (4804): 957–959. Bibcode:1987Sci...236..957S. doi:10.1126/science.236.4804.957. PMID   17812751. S2CID   42371722.