Google Books Ngram Viewer

Last updated
Example of an Ngram query Example of a google Ngram.jpg
Example of an Ngram query

The Google Books Ngram Viewer is an online search engine that charts the frequencies of any set of search strings using a yearly count of n-grams found in printed sources published between 1500 and 2022 [1] [2] [3] [4] in Google's text corpora in English, Chinese (simplified), French, German, Hebrew, Italian, Russian, or Spanish. [1] [2] [5] There are also some specialized English corpora, such as American English, British English, and English Fiction. [6]

Contents

The program can search for a word or a phrase, including misspellings or gibberish. [5] The n-grams are matched with the text within the selected corpus, and if found in 40 or more books, are then displayed as a graph. [6] The Google Books Ngram Viewer supports searches for parts of speech and wildcards. [6] It is routinely used in research. [7] [8]

History

In the development processes, Google teamed up with two Harvard researchers, Jean-Baptiste Michel and Erez Lieberman Aiden, and quietly released the program on December 16, 2010. [2] [9] Before the release, it was difficult to quantify the rate of linguistic change because of the absence of a database that was designed for this purpose, said Steven Pinker, [10] a well-known linguist who was one of the co-authors of the Science paper published on the same day. [1] The Google Books Ngram Viewer was hence developed in the hope of opening a new window to quantitative research in the humanities field, and the database contained 500 billion words from 5.2 million books publicly available from the very beginning. [2] [3] [9]

The intended audience was scholarly, but the Google Books Ngram Viewer in fact made it possible for anyone with a computer to see a graph that represents the diachronic change of the use of words and phrases with ease. Lieberman said in response to the New York Times that the developers aimed to provide even children with the ability to browse cultural trends throughout history. [9] In the Science paper, Lieberman and his collaborators called the method of high-volume data analysis in digitalized texts "culturomics". [1] [9]

Usage

Commas delimit user-entered search terms, where each comma-separated term is searched in the database as an n-gram (for example, "nursery school" is a 2-gram or bigram). [6] The Ngram Viewer then returns a plotted line chart. Note that due to limitations on the size of the Ngram database, only matches found in at least 40 books are indexed. [6]

Limitations

The data sets of the Ngram Viewer have been criticized for their reliance upon inaccurate optical character recognition (OCR) and for including large numbers of incorrectly dated and categorized texts. [11] Because of these errors, and because they are uncontrolled for bias [12] (such as the increasing amount of scientific literature, which causes other terms to appear to decline in popularity), care must be taken in using the corpora to study language or test theories. [13] Furthermore, the data sets may not reflect general linguistic or cultural change and can only hint at such an effect because they do not involve any metadata like date published,[ dubious – discuss ] author, length, or genre, to avoid any potential copyright infringements. [14]

Systemic errors like the confusion of s and f in pre-19th century texts (due to the use of ſ, the long s, which is similar in appearance to f) can cause systemic bias. [13] Although the Google Books team claims that the results are reliable from 1800 onwards, poor OCR and insufficient data mean that frequencies given for languages such as Chinese may only be accurate from 1970 onward, with earlier parts of the corpus showing no results at all for common terms, and data for some years containing more than 50% noise. [15] [16] [ better source needed ]

Guidelines for doing research with data from Google Ngram have been proposed that try to address some of the issues discussed above. [17]

See also

Related Research Articles

Corpus linguistics is an empirical method for the study of language by way of a text corpus. Corpora are balanced, often stratified collections of authentic, "real world", text of speech or writing that aim to represent a given linguistic variety. Today, corpora are generally machine-readable data collections.

<span class="mw-page-title-main">Optical character recognition</span> Computer recognition of visual text

Optical character recognition or optical character reader (OCR) is the electronic or mechanical conversion of images of typed, handwritten or printed text into machine-encoded text, whether from a scanned document, a photo of a document, a scene photo or from subtitle text superimposed on an image.

In linguistics and natural language processing, a corpus or text corpus is a dataset, consisting of natively digital and older, digitalized, language resources, either annotated or unannotated.

Text mining, text data mining (TDM) or text analytics is the process of deriving high-quality information from text. It involves "the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources." Written resources may include websites, books, emails, reviews, and articles. High-quality information is typically obtained by devising patterns and trends by means such as statistical pattern learning. According to Hotho et al. (2005) we can distinguish between three different perspectives of text mining: information extraction, data mining, and a knowledge discovery in databases (KDD) process. Text mining usually involves the process of structuring the input text, deriving patterns within the structured data, and finally evaluation and interpretation of the output. 'High quality' in text mining usually refers to some combination of relevance, novelty, and interest. Typical text mining tasks include text categorization, text clustering, concept/entity extraction, production of granular taxonomies, sentiment analysis, document summarization, and entity relation modeling.

<i>n</i>-gram Item sequences in computational linguistics

An n-gram is a sequence of n adjacent symbols in particular order. The symbols may be n adjacent letters, syllables, or rarely whole words found in a language dataset; or adjacent phonemes extracted from a speech-recording dataset, or adjacent base pairs extracted from a genome. They are collected from a text corpus or speech corpus. If Latin numerical prefixes are used, then n-gram of size 1 is called a "unigram", size 2 a "bigram" etc. If, instead of the Latin ones, the English cardinal numbers are furtherly used, then they are called "four-gram", "five-gram", etc. Similarly, using Greek numerical prefixes such as "monomer", "dimer", "trimer", "tetramer", "pentamer", etc., or English cardinal numbers, "one-mer", "two-mer", "three-mer", etc. are used in computational biology, for polymers or oligomers of a known size, called k-mers. When the items are words, n-grams may also be called shingles.

The American National Corpus (ANC) is a text corpus of American English containing 22 million words of written and spoken data produced since 1990. Currently, the ANC includes a range of genres, including emerging genres such as email, tweets, and web data that are not included in earlier corpora such as the British National Corpus. It is annotated for part of speech and lemma, shallow parse, and named entities.

Statistical machine translation (SMT) was a machine translation approach, that superseded the previous, rule-based approach because it required explicit description of each and every linguistic rule, which was costly, and which often did not generalize to other languages. Since 2003, the statistical approach itself has been gradually superseded by the deep learning-based neural machine translation.

The International Corpus of English (ICE) is a set of text corpora representing varieties of English from around the world. Over twenty countries or groups of countries where English is the first language or an official second language are included.

The Corpus of Contemporary American English (COCA) is a one-billion-word corpus of contemporary American English. It was created by Mark Davies, retired professor of corpus linguistics at Brigham Young University (BYU).

<span class="mw-page-title-main">Erez Lieberman Aiden</span> American scientist (born 1980)

Erez Lieberman Aiden is an American research scientist active in multiple fields related to applied mathematics. He is a professor of molecular and human genetics and Emeritus McNair Scholar at the Baylor College of Medicine, and formerly a fellow at the Harvard Society of Fellows and visiting faculty member at Google. He is an adjunct professor of computer science at Rice University. Using mathematical and computational approaches, he has studied evolution in a range of contexts, including that of networks through evolutionary graph theory and languages in the field of culturomics. He has published scientific articles in a variety of disciplines.

A word list is a list of a language's lexicon within some given text corpus, serving the purpose of vocabulary acquisition. A lexicon sorted by frequency "provides a rational basis for making sure that learners get the best return for their vocabulary learning effort", but is mainly intended for course writers, not directly for learners. Frequency lists are also made for lexicographical purposes, serving as a sort of checklist to ensure that common words are not left out. Some major pitfalls are the corpus content, the corpus register, and the definition of "word". While word counting is a thousand years old, with still gigantic analysis done by hand in the mid-20th century, natural language electronic processing of large corpora such as movie subtitles has accelerated the research field.

<span class="mw-page-title-main">Mark Davies (linguist)</span> American linguist (born 1963)

Mark E. Davies is an American linguist. He specializes in corpus linguistics and language variation and change. He is the creator of most of the text corpora from English-Corpora.org as well as the Corpus del español and the Corpus do português. He has also created large datasets of word frequency, collocates, and n-grams data, which have been used by many large companies in the fields of technology and also language learning.

Culturomics is a form of computational lexicology that studies human behavior and cultural trends through the quantitative analysis of digitized texts. Researchers data mine large digital archives to investigate cultural phenomena reflected in language and word usage. The term is an American neologism first described in a 2010 Science article called Quantitative Analysis of Culture Using Millions of Digitized Books, co-authored by Harvard researchers Jean-Baptiste Michel and Erez Lieberman Aiden.

The following outline is provided as an overview of and topical guide to natural-language processing:

Computational social science is an interdisciplinary academic sub-field concerned with computational approaches to the social sciences. This means that computers are used to model, simulate, and analyze social phenomena. It has been applied in areas such as computational economics, computational sociology, computational media analysis, cliodynamics, culturomics, nonprofit studies. It focuses on investigating social and behavioral relationships and interactions using data science approaches, network analysis, social simulation and studies using interactive systems.

In natural language processing (NLP), a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.

<span class="mw-page-title-main">Sketch Engine</span> Corpus manager and text analysis software

Sketch Engine is a corpus manager and text analysis software developed by Lexical Computing since 2003. Its purpose is to enable people studying language behaviour to search large text collections according to complex and linguistically motivated queries. Sketch Engine gained its name after one of the key features, word sketches: one-page, automatic, corpus-derived summaries of a word's grammatical and collocational behaviour. Currently, it supports and provides corpora in over 90 languages.

Word2vec is a technique in natural language processing (NLP) for obtaining vector representations of words. These vectors capture information about the meaning of the word based on the surrounding words. The word2vec algorithm estimates these representations by modeling text in a large corpus. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence. Word2vec was developed by Tomáš Mikolov and colleagues at Google and published in 2013.

The TenTen Corpus Family (also called TenTen corpora) is a set of comparable web text corpora, i.e. collections of texts that have been crawled from the World Wide Web and processed to match the same standards. These corpora are made available through the Sketch Engine corpus manager. There are TenTen corpora for more than 35 languages. Their target size is 10 billion (1010) words per language, which gave rise to the corpus family's name.

References

  1. 1 2 3 4 Michael, Jean-Baptiste; Shen, Yuan K.; Aiden, Aviva P.; Veres, Adrian; Gray, Matthew K.; The Google Books Team; Pickett, Joseph P.; Hoiberg, Dale; Clancy, Dan; Norvig, Peter; Orwant, Jon; Pinker, Steven; Nowak, Martin A.; Aiden, Erez L. (2010). "Quantitative Analysis of Culture Using Millions of Digitized Books". Science. 331 (6014): 176–182. doi:10.1126/science.1199644. PMC   3279742 . PMID   21163965.
  2. 1 2 3 4 Bosker, Bianca (2010-12-17). "Google Ngram Database Tracks Popularity Of 500 Billion Words". The Huffington Post . Retrieved 2012-05-31.
  3. 1 2 Lance Whitney (2010-12-17). "Google's Ngram Viewer: A time machine for wordplay". Cnet.com. Archived from the original on 2014-01-23. Retrieved 2012-05-31.
  4. ↑ @searchliaison (July 13, 2020). "The Google Books Ngram Viewer has now been updated with fresh data through 2019" (Tweet). Retrieved 2020-08-11– via Twitter.
  5. 1 2 "Google Books Ngram Viewer - University at Buffalo Libraries". Lib.Buffalo.edu. 2011-08-22. Archived from the original on 2013-07-02. Retrieved 2012-05-31.
  6. 1 2 3 4 5 "Google Books Ngram Viewer - Information" . Retrieved 2024-06-01.
  7. ↑ Greenfield, Patricia M. (2013). "The Changing Psychology of Culture From 1800 Through 2000". Psychological Science . 24 (9): 1722–1731. doi:10.1177/0956797613479387. ISSN   0956-7976. PMID   23925305. S2CID   6123553.
  8. ↑ Younes, Nadja; Reips, Ulf-Dietrich (2018). "The changing psychology of culture in German-speaking countries: A Google Ngram study". International Journal of Psychology . 53: 53–62. doi:10.1002/ijop.12428. PMID   28474338. S2CID   7440938.
  9. 1 2 3 4 "In 500 Billion Words, New Window on Culture". The New York Times. 2010-12-16. Retrieved 2024-06-01.
  10. ↑ "Steven Pinker – The Stuff of Thought: Language as a window into human nature". Royal Society of Arts. 2010-02-04. Retrieved 2024-06-02– via YouTube.
  11. ↑ Nunberg, Geoff (2010-12-16). "Humanities research with the Google Books corpus". Archived from the original on 2016-03-10. Retrieved 2015-04-19.
  12. ↑ Pechenick, Eitan Adam; Danforth, Christopher M.; Dodds, Peter Sheridan; Barrat, Alain (2015-10-07). "Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution". PLOS One . 10 (10): e0137041. arXiv: 1501.00960 . Bibcode:2015PLoSO..1037041P. doi: 10.1371/journal.pone.0137041 . PMC   4596490 . PMID   26445406.
  13. 1 2 Zhang, Sarah. "The Pitfalls of Using Google Ngram to Study Language". WIRED. Retrieved 2017-05-24.
  14. ↑ Koplenig, Alexander (2015-09-02). "The impact of lacking metadata for the measurement of cultural and linguistic change using the Google Ngram data sets—Reconstructing the composition of the German corpus in times of WWII" . Digital Scholarship in the Humanities . 32 (1). Oxford Academic (published 2017-04-01): 169–188. doi:10.1093/llc/fqv037. ISSN   2055-7671.
  15. ↑ "Google n-grams and pre-modern Chinese". digitalsinology.org. Retrieved 2015-04-19.
  16. ↑ "When n-grams go bad". digitalsinology.org. Retrieved 2015-04-19.
  17. ↑ Younes, Nadja; Reips, Ulf-Dietrich (2019-03-22). "Guideline for improving the reliability of Google Ngram studies: Evidence from religious terms". PLOS One . 14 (3): e0213554. Bibcode:2019PLoSO..1413554Y. doi: 10.1371/journal.pone.0213554 . ISSN   1932-6203. PMC   6430395 . PMID   30901329.

Bibliography

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.