Semantic folding

Last updated

Semantic folding theory describes a procedure for encoding the semantics of natural language text in a semantically grounded binary representation. This approach provides a framework for modelling how language data is processed by the neocortex. [1]

Contents

Theory

Semantic folding theory draws inspiration from Douglas R. Hofstadter's Analogy as the Core of Cognition which suggests that the brain makes sense of the world by identifying and applying analogies. [2] The theory hypothesises that semantic data must therefore be introduced to the neocortex in such a form as to allow the application of a similarity measure and offers, as a solution, the sparse binary vector employing a two-dimensional topographic semantic space as a distributional reference frame. The theory builds on the computational theory of the human cortex known as hierarchical temporal memory (HTM), and positions itself as a complementary theory for the representation of language semantics.

A particular strength claimed by this approach is that the resulting binary representation enables complex semantic operations to be performed simply and efficiently at the most basic computational level.

Two-dimensional semantic space

Analogous to the structure of the neocortex, Semantic Folding theory posits the implementation of a semantic space as a two-dimensional grid. This grid is populated by context-vectors [note 1] in such a way as to place similar context-vectors closer to each other, for instance, by using competitive learning principles. This vector space model is presented in the theory as an equivalence to the well known word space model [3] described in the information retrieval literature.

Given a semantic space (implemented as described above) a word-vector [note 2] can be obtained for any given word Y by employing the following algorithm:

For each position X in the semantic map (where X represents cartesian coordinates)     if the word Y is contained in the context-vector at position X         then add 1 to the corresponding position in the word-vector for Yelse         add 0 to the corresponding position in the word-vector for Y

The result of this process will be a word-vector containing all the contexts in which the word Y appears and will therefore be representative of the semantics of that word in the semantic space. It can be seen that the resulting word-vector is also in a sparse distributed representation (SDR) format [Schütze, 1993] & [Sahlgreen, 2006]. [3] [4] Some properties of word-SDRs that are of particular interest with respect to computational semantics are: [5]

Semantic spaces

Semantic spaces [note 3] [6] in the natural language domain aim to create representations of natural language that are capable of capturing meaning. The original motivation for semantic spaces stems from two core challenges of natural language: Vocabulary mismatch (the fact that the same meaning can be expressed in many ways) and ambiguity of natural language (the fact that the same term can have several meanings).

The application of semantic spaces in natural language processing (NLP) aims at overcoming limitations of rule-based or model-based approaches operating on the keyword level. The main drawback with these approaches is their brittleness, and the large manual effort required to create either rule-based NLP systems or training corpora for model learning. [7] [8] Rule-based and machine learning-based models are fixed on the keyword level and break down if the vocabulary differs from that defined in the rules or from the training material used for the statistical models.

Research in semantic spaces dates back more than 20 years. In 1996, two papers were published that raised a lot of attention around the general idea of creating semantic spaces: latent semantic analysis [9] from Microsoft and Hyperspace Analogue to Language [10] from the University of California. However, their adoption was limited by the large computational effort required to construct and use those semantic spaces. A breakthrough with regard to the accuracy of modelling associative relations between words (e.g. "spider-web", "lighter-cigarette", as opposed to synonymous relations such as "whale-dolphin", "astronaut-driver") was achieved by explicit semantic analysis (ESA) [11] in 2007. ESA was a novel (non-machine learning) based approach that represented words in the form of vectors with 100,000 dimensions (where each dimension represents an Article in Wikipedia). However practical applications of the approach are limited due to the large number of required dimensions in the vectors.

More recently, advances in neural networking techniques in combination with other new approaches (tensors) led to a host of new recent developments: Word2vec [12] from Google and GloVe [13] from Stanford University.

Semantic folding represents a novel, biologically inspired approach to semantic spaces where each word is represented as a sparse binary vector with 16,000 dimensions (a semantic fingerprint) in a 2D semantic map (the semantic universe). Sparse binary representation are advantageous in terms of computational efficiency, and allow for the storage of very large numbers of possible patterns. [5]

Visualization

Semantic fingerprint image comparing the terms "dog" and "car". Semantic fingerprint comparing the terms "dog" and "car".png
Semantic fingerprint image comparing the terms "dog" and "car".
Semantic fingerprint image comparing the terms "jaguar" and "Porsche" Semantic fingerprint comparing the terms "jaguar" and "Porsche".png
Semantic fingerprint image comparing the terms "jaguar" and "Porsche"

The topological distribution over a two-dimensional grid (outlined above) lends itself to a bitmap type visualization of the semantics of any word or text, where each active semantic feature can be displayed as e.g. a pixel. As can be seen in the images shown here, this representation allows for a direct visual comparison of the semantics of two (or more) linguistic items.

Image 1 clearly demonstrates that the two disparate terms "dog" and "car" have, as expected, very obviously different semantics.

Image 2 shows that only one of the meaning contexts of "jaguar", that of "Jaguar" the car, overlaps with the meaning of Porsche (indicating partial similarity). Other meaning contexts of "jaguar" e.g. "jaguar" the animal clearly have different non-overlapping contexts. The visualization of semantic similarity using Semantic Folding bears a strong resemblance to the fMRI images produced in a research study conducted by A.G. Huth et al., [14] [15] where it is claimed that words are grouped in the brain by meaning. voxels, little volume segments of the brain, were found to follow a pattern were semantic information is represented along the boundary of the visual cortex with visual and linguistic categories represented on posterior and anterior side respectively. [16] [17] [18]

Notes

  1. ↑ A context-vector is defined as a vector containing all the words in a particular context.
  2. ↑ A word-vector or word-SDR is referred to as a Semantic Fingerprint in Semantic Folding theory.
  3. ↑ also referred to as distributed semantic spaces or distributed semantic memory

Related Research Articles

<span class="mw-page-title-main">Semantic network</span> Knowledge base that represents semantic relations between concepts in a network

A semantic network, or frame network is a knowledge base that represents semantic relations between concepts in a network. This is often used as a form of knowledge representation. It is a directed or undirected graph consisting of vertices, which represent concepts, and edges, which represent semantic relations between concepts, mapping or connecting semantic fields. A semantic network may be instantiated as, for example, a graph database or a concept map. Typical standardized semantic networks are expressed as semantic triples.

Word-sense disambiguation is the process of identifying which sense of a word is meant in a sentence or other segment of context. In human language processing and cognition, it is usually subconscious.

David Courtenay Marr was a British neuroscientist and physiologist. Marr integrated results from psychology, artificial intelligence, and neurophysiology into new models of visual processing. His work was influential in computational neuroscience and led to a resurgence of interest in the discipline.

Semantic memory refers to general world knowledge that humans have accumulated throughout their lives. This general knowledge is intertwined in experience and dependent on culture. New concepts are learned by applying knowledge learned from things in the past.

Latent semantic analysis (LSA) is a technique in natural language processing, in particular distributional semantics, of analyzing relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. LSA assumes that words that are close in meaning will occur in similar pieces of text. A matrix containing word counts per document is constructed from a large piece of text and a mathematical technique called singular value decomposition (SVD) is used to reduce the number of rows while preserving the similarity structure among columns. Documents are then compared by cosine similarity between any two columns. Values close to 1 represent very similar documents while values close to 0 represent very dissimilar documents.

Semantic similarity is a metric defined over a set of documents or terms, where the idea of distance between items is based on the likeness of their meaning or semantic content as opposed to lexicographical similarity. These are mathematical tools used to estimate the strength of the semantic relationship between units of language, concepts or instances, through a numerical description obtained according to the comparison of information supporting their meaning or describing their nature. The term semantic similarity is often confused with semantic relatedness. Semantic relatedness includes any relation between two terms, while semantic similarity only includes "is a" relations. For example, "car" is similar to "bus", but is also related to "road" and "driving".

Neurophilosophy or the philosophy of neuroscience is the interdisciplinary study of neuroscience and philosophy that explores the relevance of neuroscientific studies to the arguments traditionally categorized as philosophy of mind. The philosophy of neuroscience attempts to clarify neuroscientific methods and results using the conceptual rigor and methods of philosophy of science.

In linguistics, statistical semantics applies the methods of statistics to the problem of determining the meaning of words or phrases, ideally through unsupervised learning, to a degree of precision at least sufficient for the purpose of information retrieval.

<span class="mw-page-title-main">Distributional semantics</span> Field of linguistics

Distributional semantics is a research area that develops and studies theories and methods for quantifying and categorizing semantic similarities between linguistic items based on their distributional properties in large samples of language data. The basic idea of distributional semantics can be summed up in the so-called distributional hypothesis: linguistic items with similar distributions have similar meanings.

Hierarchical temporal memory (HTM) is a biologically constrained machine intelligence technology developed by Numenta. Originally described in the 2004 book On Intelligence by Jeff Hawkins with Sandra Blakeslee, HTM is primarily used today for anomaly detection in streaming data. The technology is based on neuroscience and the physiology and interaction of pyramidal neurons in the neocortex of the mammalian brain.

Sparse distributed memory (SDM) is a mathematical model of human long-term memory introduced by Pentti Kanerva in 1988 while he was at NASA Ames Research Center.

Pentti Kanerva is an American neuroscientist and the originator of the sparse distributed memory model. He is responsible for relating the properties of long-term memory to mathematical properties of high-dimensional spaces and compares artificial neural-net associative memory to conventional computer random-access memory and to the neurons in the brain. This theory has been applied to design and implement the random indexing approach to learning semantic relations from linguistic data. Kanerva was also the first to use a computer clipboard to preserve deleted texts. The operation would later come to be known as cut, copy, and paste.

Random indexing is a dimensionality reduction method and computational framework for distributional semantics, based on the insight that very-high-dimensional vector space model implementations are impractical, that models need not grow in dimensionality when new items are encountered, and that a high-dimensional model can be projected into a space of lower dimensionality without compromising L2 distance metrics if the resulting dimensions are chosen appropriately.

The following outline is provided as an overview of and topical guide to natural-language processing:

In natural language processing, a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.

Word2vec is a technique in natural language processing (NLP) for obtaining vector representations of words. These vectors capture information about the meaning of the word based on the surrounding words. The word2vec algorithm estimates these representations by modeling text in a large corpus. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence. Word2vec was developed by Tomáš Mikolov and colleagues at Google and published in 2013.

A distributional–relational database, or word-vector database, is a database management system (DBMS) that uses distributional word-vector representations to enrich the semantics of structured data.

Semantic spaces in the natural language domain aim to create representations of natural language that are capable of capturing meaning. The original motivation for semantic spaces stems from two core challenges of natural language: Vocabulary mismatch and ambiguity of natural language.

<span class="mw-page-title-main">Semantic parsing</span>

Semantic parsing is the task of converting a natural language utterance to a logical form: a machine-understandable representation of its meaning. Semantic parsing can thus be understood as extracting the precise meaning of an utterance. Applications of semantic parsing include machine translation, question answering, ontology induction, automated reasoning, and code generation. The phrase was first used in the 1970s by Yorick Wilks as the basis for machine translation programs working with only semantic representations. Semantic parsing is one of the important tasks in computational linguistics and natural language processing.

The usage-based linguistics is a linguistics approach within a broader functional/cognitive framework, that emerged since the late 1980s, and that assumes a profound relation between linguistic structure and usage. It challenges the dominant focus, in 20th century linguistics, on considering language as an isolated system removed from its use in human interaction and human cognition. Rather, usage-based models posit that linguistic information is expressed via context-sensitive mental processing and mental representations, which have the cognitive ability to succinctly account for the complexity of actual language use at all levels. Broadly speaking, a usage-based model of language accounts for language acquisition and processing, synchronic and diachronic patterns, and both low-level and high-level structure in language, by looking at actual language use.

References

  1. ↑ De Sousa Webber, Francisco (2015). "Semantic Folding theory and its Application in Semantic Fingerprinting". Cornell University Library. arXiv: 1511.08855 . Bibcode:2015arXiv151108855D.
  2. ↑ The Analogical Mind. A Bradford Book. 2 March 2001. ISBN   9780262072069 . Retrieved 2016-04-18.{{cite book}}: |website= ignored (help)
  3. 1 2 Sahlgreen, Magnus (2006). "The Word-Space Model".
  4. ↑ Schütze, Hinrich (1993). "Word Space": 895–902. CiteSeerX   10.1.1.41.8856 .{{cite journal}}: Cite journal requires |journal= (help)
  5. 1 2 Subutai Ahmad; Jeff Hawkins (2015). "Properties of Sparse Distributed Representations and their Application to Hierarchical Temporal Memory". arXiv: 1503.07469 [q-bio.NC].
  6. ↑ Baroni, Marco; Lenci, Alessandro (2010). "Distributional Memory: A General Framework for Corpus-Based Semantics". Computational Linguistics. 36 (4): 673–721. CiteSeerX   10.1.1.331.3769 . doi:10.1162/coli_a_00016. S2CID   5584134.
  7. ↑ Scott C. Deerwester; Susan T. Dumais; Thomas K. Landauer; George W. Furnas; Richard A. Harshen (1990). "Indexing by Latent Semantic Analysis" (PDF). Journal of the American Society for Information Science.
  8. ↑ Xing Wei; W. Bruce Croft (2007). "Investigating retrieval performance with manually-built topic models". Proceeding RIAO '07 Large Scale Semantic Access to Content (Text, Image, Video, and Sound). Riao '07: 333–349.
  9. ↑ "LSA: A Solution to Plato's Problem". lsa.colorado.edu. Retrieved 2016-04-19.
  10. ↑ Lund, Kevin; Burgess, Curt (1996-06-01). "Producing high-dimensional semantic spaces from lexical co-occurrence". Behavior Research Methods, Instruments, & Computers. 28 (2): 203–208. doi: 10.3758/BF03204766 . ISSN   0743-3808.
  11. ↑ Evgeniy Gabrilovich & Shaul Markovitch (2007). "Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis" (PDF). Proc. 20th Int'l Joint Conf. On Artificial Intelligence (IJCAI). Pp. 1606–1611.
  12. ↑ Tomas Mikolov; Ilya Sutskever; Kai Chen; Greg Corrado; Jeffrey Dean (2013). "Distributed Representations of Words and Phrases and their Compositionality". arXiv: 1310.4546 [cs.CL].
  13. ↑ Jeffrey Pennington; Richard Socher; Christopher D. Manning (2014). "GloVe: Global Vectors for Word Representation" (PDF).
  14. ↑ Huth, Alexander (27 April 2016). "Natural speech reveals the semantic maps that tile human cerebral cortex". Nature. 532 (7600): 453–458. Bibcode:2016Natur.532..453H. doi:10.1038/nature17637. PMC   4852309 . PMID   27121839.
  15. ↑ "Brain". gallantlab.org. Retrieved 2022-02-16.
  16. ↑ Popham, Sara F.; Huth, Alexander G.; Bilenko, Natalia Y.; Deniz, Fatma; Gao, James S.; Nunez-Elizalde, Anwar O.; Gallant, Jack L. (11 August 2021). "Visual and linguistic semantic representations are aligned at the border of human visual cortex". Nature Neuroscience. 24 (11): 1628–1636. doi:10.1038/s41593-021-00921-6. ISSN   1097-6256. PMID   34711960. S2CID   240152854.
  17. ↑ Steel, Adam; Billings, Madeleine M.; Silson, Edward H.; Robertson, Caroline E. (2021-05-11). "A network linking scene perception and spatial memory systems in posterior cerebral cortex". Nature Communications. 12 (1): 2632. Bibcode:2021NatCo..12.2632S. doi:10.1038/s41467-021-22848-z. ISSN   2041-1723. PMC   8113503 . PMID   33976141.
  18. ↑ Cepelewicz, Jordana (2022-02-08). "New Map of Meaning in the Brain Changes Ideas About Memory". Quanta Magazine. Retrieved 2022-02-16.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.