Representational harm

Last updated

Systems cause representational harm when they misrepresent a group of people in a negative manner. Representational harms include perpetuating harmful stereotypes about or minimizing the existence of a social group, such as a racial, ethnic, gender, or religious group. [1] Machine learning algorithms often commit representational harm when they learn patterns from data that have built-in biases. While preventing representational harm in models is essential to prevent harmful biases, researchers often lack precise definitions of representational harm and conflate it with allocative harm, an unequal distribution of resources among social groups, which is more widely studied and easier to measure. [1] However, recognition of representational harms is growing and preventing them has become an active research area. Researchers have recently developed methods to effectively quantify representational harm in algorithms, making progress on preventing this harm in the future. [2] [3]

Contents

Types

Three prominent types of representational harm include stereotyping, denigration, and misrecognition. [4] These subcategories present many dangers to individuals and groups.

Stereotypes are oversimplified and usually undesirable representations of a specific group of people, usually by race and gender. This often leads to the denial of educational, employment, housing, and other opportunities. [5] For example, the minority stereotype of Asian Americans as highly intelligent and good at mathematics can be damaging professionally and academically. [6]

Denigration is the action of unfairly criticizing individuals. This frequently happens when the demeaning of social groups occurs. [5] For example, when searching for "Black-sounding" names versus "white-sounding" ones, some retrieval systems bolster the false perception of criminality by displaying ads for bail-bonding businesses. [7] A system may shift the representation of a group to be of lower social status, often resulting in a disregard from society. [5]

Misrecognition, or incorrect recognition, can display in many forms, including, but not limited to, erasing and alienating social groups, and denying people the right to self-identify. [5] Erasing and alienating social groups involves the unequal visibility of certain social groups; specifically, systematic ineligibility in algorithmic systems perpetuates inequality by contributing to the underrepresentation of social groups. [5] Not allowing people to self-identify is closely related as people's identities can be ‘erased’ or ‘alienated’ in these algorithms. Misrecognition causes more than surface-level harm to individuals: psychological harm, social isolation, and emotional insecurity can emerge from this subcategory of representational harm. [5]

Quantification

As the dangers of representational harm have become better understood, some researchers have developed methods to measure representational harm in algorithms.

Modeling stereotyping is one way to identify representational harm. Representational stereotyping can be quantified by comparing the predicted outcomes for one social group with the ground-truth outcomes for that group observed in real data. [2] For example, if individuals from group A achieve an outcome with a probability of 60%, stereotyping would be observed if it predicted individuals to achieve that outcome with a probability greater than 60%. [2] The group modeled stereotyping in the context of classification, regression, and clustering problems, and developed a set of rules to quantitatively determine if the model predictions exhibit stereotyping in each of these cases.[ citation needed ]

Other attempts to measure representational harms have focused on applications of algorithms in specific domains such as image captioning, the act of an algorithm generating a short description of an image. In a study on image captioning, researchers measured five types of representational harm. To quantify stereotyping, they measured the number of incorrect words included in the model-generated image caption when compared to a gold-standard caption. [3] They manually reviewed each of the incorrectly included words, determining whether the incorrect word reflected a stereotype associated with the image or whether it was an unrelated error, which allowed them to have a proxy measure of the amount of stereotyping occurring in this caption generation. [3] These researchers also attempted to measure demeaning representational harm. To measure this, they analyzed the frequency with which humans in the image were mentioned in the generated caption. It was hypothesized that if the individuals were not mentioned in the caption, then this was a form of dehumanization. [3]

Examples

One of the most notorious examples of representational harm was committed by Google in 2015 when an algorithm in Google Photos classified Black people as gorillas. [8] Developers at Google said that the problem was caused because there were not enough faces of Black people in the training dataset for the algorithm to learn the difference between Black people and gorillas. [9] Google issued an apology and fixed the issue by blocking its algorithms from classifying anything as a primate. [9] In 2023, Google's photos algorithm was still blocked from identifying gorillas in photos. [9]

Another prevalent example of representational harm is the possibility of stereotypes being encoded in word embeddings, which are trained using a wide range of text. These word embeddings are the representation of a word as an array of numbers in vector space, which allows an individual to calculate the relationships and similarities between words. [10] However, recent studies have shown that these word embeddings may commonly encode harmful stereotypes, such as the common example that the phrase “computer programmer” is oftentimes more closely related to “man” than it is to “women” in vector space. [11] This could be interpreted as a misrepresentation of computer programming as a profession that is better performed by men, which would be an example of representational harm.

Related Research Articles

Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data, and thus perform tasks without explicit instructions. Recently, generative artificial neural networks have been able to surpass many previous approaches in performance.

Analytics is the systematic computational analysis of data or statistics. It is used for the discovery, interpretation, and communication of meaningful patterns in data. It also entails applying data patterns toward effective decision-making. It can be valuable in areas rich with recorded information; analytics relies on the simultaneous application of statistics, computer programming, and operations research to quantify performance.

Semantic similarity is a metric defined over a set of documents or terms, where the idea of distance between items is based on the likeness of their meaning or semantic content as opposed to lexicographical similarity. These are mathematical tools used to estimate the strength of the semantic relationship between units of language, concepts or instances, through a numerical description obtained according to the comparison of information supporting their meaning or describing their nature. The term semantic similarity is often confused with semantic relatedness. Semantic relatedness includes any relation between two terms, while semantic similarity only includes "is a" relations. For example, "car" is similar to "bus", but is also related to "road" and "driving".

The implicit-association test (IAT) is an assessment intended to detect subconscious associations between mental representations of objects (concepts) in memory. Its best-known application is the assessment of implicit stereotypes held by test subjects, such as associations between particular racial categories and stereotypes about those groups. The test has been applied to a variety of belief associations, such as those involving racial groups, gender, sexuality, age, and religion but also the self-esteem, political views, and predictions of the test taker. The implicit-association test is the subject of significant academic and popular debate regarding its validity, reliability, and usefulness in assessing implicit bias.

Social stigma is the disapproval of, or discrimination against, an individual or group based on perceived characteristics that serve to distinguish them from other members of a society. Social stigmas are commonly related to culture, gender, race, socioeconomic class, age, sexual orientation, sexuality, body image, physical disability, intelligence or lack thereof, and health. Some stigma may be obvious, while others are known as concealable stigmas that must be revealed through disclosure. Stigma can also be against oneself, stemming from negatively viewed personal attributes in a way that can result in a "spoiled identity".

System justification theory is a theory within social psychology that system-justifying beliefs serve a psychologically palliative function. It proposes that people have several underlying needs, which vary from individual to individual, that can be satisfied by the defense and justification of the status quo, even when the system may be disadvantageous to certain people. People have epistemic, existential, and relational needs that are met by and manifest as ideological support for the prevailing structure of social, economic, and political norms. Need for order and stability, and thus resistance to change or alternatives, for example, can be a motivator for individuals to see the status quo as good, legitimate, and even desirable.

Dynamic network analysis (DNA) is an emergent scientific field that brings together traditional social network analysis (SNA), link analysis (LA), social simulation and multi-agent systems (MAS) within network science and network theory. Dynamic networks are a function of time to a set of graphs; for each time point there is a graph. This is akin to the definition of dynamical systems, in which the function is from time to an ambient space, where instead of ambient space time is translated to relationships between pairs of vertices.

Social search is a behavior of retrieving and searching on a social searching engine that mainly searches user-generated content such as news, videos and images related search queries on social media like Facebook, LinkedIn, Twitter, Instagram and Flickr. It is an enhanced version of web search that combines traditional algorithms. The idea behind social search is that instead of ranking search results purely based on semantic relevance between a query and the results, a social search system also takes into account social relationships between the results and the searcher. The social relationships could be in various forms. For example, in LinkedIn people search engine, the social relationships include social connections between searcher and each result, whether or not they are in the same industries, work for the same companies, belong the same social groups, and go the same schools, etc.

Implicit attitudes are evaluations that occur without conscious awareness towards an attitude object or the self. These evaluations are generally either favorable or unfavorable and come about from various influences in the individual experience. The commonly used definition of implicit attitude within cognitive and social psychology comes from Anthony Greenwald and Mahzarin Banaji's template for definitions of terms related to implicit cognition: "Implicit attitudes are introspectively unidentified traces of past experience that mediate favorable or unfavorable feeling, thought, or action toward social objects". These thoughts, feelings or actions have an influence on behavior that the individual may not be aware of.

Personalized search is a web search tailored specifically to an individual's interests by incorporating information about the individual beyond the specific query provided. There are two general approaches to personalizing search results, involving modifying the user's query and re-ranking search results.

In statistics and natural language processing, a topic model is a type of statistical model for discovering the abstract "topics" that occur in a collection of documents. Topic modeling is a frequently used text-mining tool for discovery of hidden semantic structures in a text body. Intuitively, given that a document is about a particular topic, one would expect particular words to appear in the document more or less frequently: "dog" and "bone" will appear more often in documents about dogs, "cat" and "meow" will appear in documents about cats, and "the" and "is" will appear approximately equally in both. A document typically concerns multiple topics in different proportions; thus, in a document that is 10% about cats and 90% about dogs, there would probably be about 9 times more dog words than cat words. The "topics" produced by topic modeling techniques are clusters of similar words. A topic model captures this intuition in a mathematical framework, which allows examining a set of documents and discovering, based on the statistics of the words in each, what the topics might be and what each document's balance of topics is.

<span class="mw-page-title-main">Filter bubble</span> Intellectual isolation involving search engines

A filter bubble or ideological frame is a state of intellectual isolation that can result from personalized searches, recommendation systems, and algorithmic curation. The search results are based on information about the user, such as their location, past click-behavior, and search history. Consequently, users become separated from information that disagrees with their viewpoints, effectively isolating them in their own cultural or ideological bubbles, resulting in a limited and customized view of the world. The choices made by these algorithms are only sometimes transparent. Prime examples include Google Personalized Search results and Facebook's personalized news-stream.

An implicit bias or implicit stereotype is the pre-reflective attribution of particular qualities by an individual to a member of some social out group.

In natural language processing (NLP), a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.

DeepFace is a deep learning facial recognition system created by a research group at Facebook. It identifies human faces in digital images. The program employs a nine-layer neural network with over 120 million connection weights and was trained on four million images uploaded by Facebook users. The Facebook Research team has stated that the DeepFace method reaches an accuracy of 97.35% ± 0.25% on Labeled Faces in the Wild (LFW) data set where human beings have 97.53%. This means that DeepFace is sometimes more successful than human beings. As a result of growing societal concerns Meta announced that it plans to shut down Facebook facial recognition system, deleting the face scan data of more than one billion users. This change will represent one of the largest shifts in facial recognition usage in the technology's history. Facebook planned to delete by December 2021 more than one billion facial recognition templates, which are digital scans of facial features. However, it did not plan to eliminate DeepFace which is the software that powers the facial recognition system. The company has also not ruled out incorporating facial recognition technology into future products, according to Meta spokesperson.

<span class="mw-page-title-main">Algorithmic bias</span> Technological phenomenon with social implications

Algorithmic bias describes systematic and repeatable errors in a computer system that create "unfair" outcomes, such as "privileging" one category over another in ways different from the intended function of the algorithm.

Crowdsource is a crowdsourcing platform developed by Google intended to improve a host of Google services through the user-facing training of different algorithms.

Political bias is a bias or perceived bias involving the slanting or altering of information to make a political position or political candidate seem more attractive. With a distinct association with media bias, it commonly refers to how a reporter, news organisation, or TV show covers a political candidate or a policy issue.

Fairness in machine learning refers to the various attempts at correcting algorithmic bias in automated decision processes based on machine learning models. Decisions made by computers after a machine-learning process may be considered unfair if they were based on variables considered sensitive. For example gender, ethnicity, sexual orientation or disability. As it is the case with many ethical concepts, definitions of fairness and bias are always controversial. In general, fairness and bias are considered relevant when the decision process impacts people's lives. In machine learning, the problem of algorithmic bias is well known and well studied. Outcomes may be skewed by a range of factors and thus might be considered unfair with respect to certain groups or individuals. An example would be the way social media sites deliver personalized news to consumers.

Automated decision-making (ADM) involves the use of data, machines and algorithms to make decisions in a range of contexts, including public administration, business, health, education, law, employment, transport, media and entertainment, with varying degrees of human oversight or intervention. ADM involves large-scale data from a range of sources, such as databases, text, social media, sensors, images or speech, that is processed using various technologies including computer software, algorithms, machine learning, natural language processing, artificial intelligence, augmented intelligence and robotics. The increasing use of automated decision-making systems (ADMS) across a range of contexts presents many benefits and challenges to human society requiring consideration of the technical, legal, ethical, societal, educational, economic and health consequences.

References

  1. 1 2 Blodgett, Su Lin (2021-04-06). Sociolinguistically Driven Approaches for Just Natural Language Processing. Doctoral Dissertations (Thesis). doi:10.7275/20410631.
  2. 1 2 3 Abbasi, Mohsen; Friedler, Sorelle; Scheidegger, Carlos; Venkatasubramanian, Suresh (28 January 2019). "Fairness in representation: quantifying stereotyping as representational harm". arXiv: 1901.09565 [cs.LG].
  3. 1 2 3 4 Wang, Angelina; Barocas, Solon; Laird, Kristen; Wallach, Hanna (2022-06-20). "Measuring Representational Harms in Image Captioning". 2022 ACM Conference on Fairness, Accountability, and Transparency. FAccT '22. New York, NY, USA: Association for Computing Machinery. pp. 324–335. doi:10.1145/3531146.3533099. ISBN   978-1-4503-9352-2. S2CID   249674329.
  4. ↑ Rusanen, Anna-Mari; Nurminen, Jukka K. "Ethics of Ai". ethics-of-ai.mooc.fi.
  5. 1 2 3 4 5 6 Shelby, Renee; Rismani, Shalaleh; Henne, Kathryn; Moon, AJung; Rostamzadeh, Negar; Nicholas, Paul; Yilla-Akbari, N'Mah; Gallegos, Jess; Smart, Andrew; Garcia, Emilio; Virk, Gurleen (2023-08-29). "Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction". Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. AIES '23. New York, NY, USA: Association for Computing Machinery. pp. 723–741. doi:10.1145/3600211.3604673. ISBN   979-8-4007-0231-0. S2CID   256697294.
  6. ↑ Trytten, Deborah A.; Lowe, Anna Wong; Walden, Susan E. (January 2, 2013). ""Asians are Good at Math. What an Awful Stereotype" The Model Minority Stereotype's Impact on Asian American Engineering Students". Journal of Engineering Education. 101 (3): 439–468. doi: 10.1002/j.2168-9830.2012.tb00057.x . ISSN   1069-4730. S2CID   144783391.
  7. ↑ Sweeney, Latanya (2013-03-01). "Discrimination in Online Ad Delivery: Google ads, black names and white names, racial discrimination, and click advertising". Queue. 11 (3): 10–29. arXiv: 1301.6822 . doi: 10.1145/2460276.2460278 . ISSN   1542-7730. S2CID   35894627.
  8. ↑ "Google apologises for Photos app's racist blunder". BBC News. 2015-07-01. Retrieved 2023-12-06.
  9. 1 2 3 Grant, Nico; Hill (May 22, 2023). "Google's Photo App Still Can't Find Gorillas. And Neither Can Apple's". The New York Times . Retrieved December 5, 2023.
  10. ↑ Major, Vincent; Surkis, Alisa; Aphinyanaphongs, Yindalon (2018). "Utility of General and Specific Word Embeddings for Classifying Translational Stages of Research". AMIA ... Annual Symposium Proceedings. AMIA Symposium. 2018: 1405–1414. ISSN   1942-597X. PMC   6371342 . PMID   30815185.
  11. ↑ Bolukbasi, Tolga; Chang, Kai-Wei; Zou, James; Saligrama, Venkatesh; Kalai, Adam (21 Jul 2016). "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings". arXiv: 1607.06520 [cs.CL].
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.