Thought vector

Last updated

Thought vector is a term popularized by Geoffrey Hinton, the prominent deep-learning researcher now at Google, which uses vectors based on natural language [1] to improve its search results. [2]

Geoffrey Hinton British-Canadian computer scientist and psychologist

Geoffrey Everest Hinton is an English Canadian cognitive psychologist and computer scientist, most noted for his work on artificial neural networks. Since 2013 he divides his time working for Google and the University of Toronto.

Deep learning Branch of machine learning

Deep learning is part of a broader family of machine learning methods based on artificial neural networks. Learning can be supervised, semi-supervised or unsupervised.

Natural language processing field of computer science and linguistics

Not to be confused with Non-linear programming

Related Research Articles

Boltzmann machine Stochastic recurrent neural network

A Boltzmann machine is a type of stochastic recurrent neural network and Markov random field.

Backpropagation optimization algorithm for artificial neural networks

Backpropagation algorithms are a family of methods used to efficiently train artificial neural networks (ANNs) following a gradient-based optimization algorithm that exploits the chain rule. The main feature of backpropagation is its iterative, recursive and efficient method for calculating the weights updates to improve the network until it is able to perform the task for which it is being trained. It is closely related to the Gauss–Newton algorithm.

Peter Dayan American mathematician, author, and computer scientist

Peter Samuel Dayan is director at the Max Planck Institute for Biological Cybernetics in Tübingen, Germany. He is co-author of Theoretical Neuroscience, an influential textbook on computational neuroscience. He is known for applying bayesian methods from machine learning and artificial intelligence to understand neural function and is particularly recognized for relating neurotransmitter levels to prediction errors and Bayesian uncertainties. He has pioneered the field of reinforcement learning (RL) where he helped develop the Q-learning algorithm, and made contributions to unsupervised learning, including the wake-sleep algorithm for neural networks and the Helmholtz machine.


An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learnt, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, hence its name. Several variants exist to the basic model, with the aim of forcing the learned representations of the input to assume useful properties. Examples are the regularized autoencoders, proven effective in learning representations for subsequent classification tasks, and Variational autoencoders, with their recent applications as generative models. Autoencoders are effectively used for solving many applied problems, from face recognition to acquiring the semantic meaning for the words.

Long short-term memory recurrent neural network architecture

Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. It can not only process single data points, but also entire sequences of data. For example, LSTM is applicable to tasks such as unsegmented, connected handwriting recognition or speech recognition. Bloomberg Business Week wrote: "These powers make LSTM arguably the most commercial AI achievement, used for everything from predicting diseases to composing music."

Andrew Ng American artificial intelligence researcher

Andrew Yan-Tak Ng is a Chinese-American computer scientist and statistician, focusing on machine learning and AI. Also a business executive and investor in the Silicon Valley, Ng co-founded and led Google Brain and was a former Vice President and Chief Scientist at Baidu, building the company's Artificial Intelligence Group into a team of several thousand people.


Encog is a machine learning framework available for Java and .Net. Encog supports different learning algorithms such as Bayesian Networks, Hidden Markov Models and Support Vector Machines. However, its main strength lies in its neural network algorithms. Encog contains classes to create a wide variety of networks, as well as support classes to normalize and process data for these neural networks. Encog trains using many different techniques. Multithreading is used to allow optimal training performance on multicore machines.

Theano is a Python library and optimizing compiler for manipulating and evaluating mathematical expressions, especially matrix-valued ones. In Theano, computations are expressed using a NumPy-esque syntax and compiled to run efficiently on either CPU or GPU architectures.

Restricted Boltzmann machine artificial neural network

A restricted Boltzmann machine (RBM) is a generative stochastic artificial neural network that can learn a probability distribution over its set of inputs.

Deep belief network

In machine learning, a deep belief network (DBN) is a generative graphical model, or alternatively a class of deep neural network, composed of multiple layers of latent variables, with connections between the layers but not between units within each layer.

Deeplearning4j open source deep learning library

Eclipse Deeplearning4j is a deep learning programming library written for Java and the Java virtual machine (JVM) and a computing framework with wide support for deep learning algorithms. Deeplearning4j includes implementations of the restricted Boltzmann machine, deep belief net, deep autoencoder, stacked denoising autoencoder and recursive neural tensor network, word2vec, doc2vec, and GloVe. These algorithms all include distributed parallel versions that integrate with Apache Hadoop and Spark.

Word embedding method in natural language processing

Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with many dimensions per word to a continuous vector space with a much lower dimension.

Adam Gibson is an American entrepreneur and computer scientist. He co-founded the artificial intelligence/machine learning company Skymind and the open source frameworks Deeplearning4j and ND4J, or n-dimensional arrays for Java. He is also advisor to the data science master's program at GalvanizeU in San Francisco.

Yoshua Bengio Canadian computer scientist

Yoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning. He was a co-recipient of the 2018 ACM A.M. Turing Award for his work in deep learning. He is a professor at the Department of Computer Science and Operations Research at the Université de Montréal and scientific director of the Montreal Institute for Learning Algorithms (MILA).

The following table compares notable software frameworks, libraries and computer programs for deep learning.

ND4J is a scientific computing and linear algebra library, written in the programming language Java, operating on the Java virtual machine (JVM), and compatible with other languages such as Scala, and Clojure. ND4J was contributed to the Eclipse Foundation in October 2017.

Caffe (software) deep learning framework

CAFFE is a deep learning framework, originally developed at University of California, Berkeley. It is open source, under a BSD license. It is written in C++, with a Python interface.

A Capsule Neural Network (CapsNet) is a machine learning system that is a type of artificial neural network (ANN) that can be used to better model hierarchical relationships. The approach is an attempt to more closely mimic biological neural organization.


  1. Hinton, Geoffrey. "Aetherial Symbols" . Retrieved 2017-10-09.
  2. Gibson, Chris Nicholson, Adam. "Thought Vectors, Deep Learning & the Future of AI - Deeplearning4j: Open-source, distributed deep learning for the JVM". Archived from the original on 2017-02-11. Retrieved 2016-08-23.