Torch (machine learning)

Last updated
Torch
Torch 2014 logo.png
Original author(s) Ronan Collobert, Samy Bengio, Johnny Mariéthoz [1]
Initial releaseOctober 2002;18 years ago (2002-10) [1]
Stable release
7.0 / February 27, 2017;3 years ago (2017-02-27) [2]
Repository OOjs UI icon edit-ltr-progressive.svg
Written in Lua, LuaJIT, C, CUDA and C++
Operating system Linux, Android, Mac OS X, iOS
Type Library for machine learning and deep learning
License BSD License
Website torch.ch

Torch is an open-source machine learning library, a scientific computing framework, and a script language based on the Lua programming language. [3] It provides a wide range of algorithms for deep learning, and uses the scripting language LuaJIT, and an underlying C implementation. As of 2018, Torch is no longer in active development. [4] However PyTorch, which is based on the Torch library, is actively developed as of December 2020. [5]

Contents

torch

The core package of Torch is torch. It provides a flexible N-dimensional array or Tensor, which supports basic routines for indexing, slicing, transposing, type-casting, resizing, sharing storage and cloning. This object is used by most other packages and thus forms the core object of the library. The Tensor also supports mathematical operations like max, min, sum, statistical distributions like uniform, normal and multinomial, and BLAS operations like dot product, matrix-vector multiplication, matrix-matrix multiplication, matrix-vector product and matrix product.

The following exemplifies using torch via its REPL interpreter:

>a=torch.randn(3,4)>=a-0.2381-0.3401-1.7844-0.26150.14111.62490.17080.8299-1.04342.22911.05250.8465[torch.DoubleTensorofdimension3x4]>a[1][2]-0.34010116549482>a:narrow(1,1,2)-0.2381-0.3401-1.7844-0.26150.14111.62490.17080.8299[torch.DoubleTensorofdimension2x4]>a:index(1,torch.LongTensor{1,2})-0.2381-0.3401-1.7844-0.26150.14111.62490.17080.8299[torch.DoubleTensorofdimension2x4]>a:min()-1.7844365427828

The torch package also simplifies object oriented programming and serialization by providing various convenience functions which are used throughout its packages. The torch.class(classname, parentclass) function can be used to create object factories (classes). When the constructor is called, torch initializes and sets a Lua table with the user-defined metatable, which makes the table an object.

Objects created with the torch factory can also be serialized, as long as they do not contain references to objects that cannot be serialized, such as Lua coroutines, and Lua userdata. However, userdata can be serialized if it is wrapped by a table (or metatable) that provides read() and write() methods.

nn

The nn package is used for building neural networks. It is divided into modular objects that share a common Module interface. Modules have a forward() and backward() method that allow them to feedforward and backpropagate, respectively. Modules can be joined together using module composites, like Sequential, Parallel and Concat to create complex task-tailored graphs. Simpler modules like Linear, Tanh and Max make up the basic component modules. This modular interface provides first-order automatic gradient differentiation. What follows is an example use-case for building a multilayer perceptron using Modules:

>mlp=nn.Sequential()>mlp:add(nn.Linear(10,25))-- 10 input, 25 hidden units>mlp:add(nn.Tanh())-- some hyperbolic tangent transfer function>mlp:add(nn.Linear(25,1))-- 1 output>=mlp:forward(torch.randn(10))-0.1815[torch.Tensorofdimension1]

Loss functions are implemented as sub-classes of Criterion, which has a similar interface to Module. It also has forward() and backward() methods for computing the loss and backpropagating gradients, respectively. Criteria are helpful to train neural network on classical tasks. Common criteria are the Mean Squared Error criterion implemented in MSECriterion and the cross-entropy criterion implemented in ClassNLLCriterion. What follows is an example of a Lua function that can be iteratively called to train an mlp Module on input Tensor x, target Tensor y with a scalar learningRate:

functiongradUpdate(mlp,x,y,learningRate)localcriterion=nn.ClassNLLCriterion()pred=mlp:forward(x)localerr=criterion:forward(pred,y);mlp:zeroGradParameters();localt=criterion:backward(pred,y);mlp:backward(x,t);mlp:updateParameters(learningRate);end

It also has StochasticGradient class for training a neural network using Stochastic gradient descent, although the optim package provides much more options in this respect, like momentum and weight decay regularization.

Other packages

Many packages other than the above official packages are used with Torch. These are listed in the torch cheatsheet. [6] These extra packages provide a wide range of utilities such as parallelism, asynchronous input/output, image processing, and so on. They can be installed with LuaRocks, the Lua package manager which is also included with the Torch distribution.

Applications

Torch is used by the Facebook AI Research Group, [7] IBM, [8] Yandex [9] and the Idiap Research Institute. [10] Torch has been extended for use on Android [11] and iOS. [12] It has been used to build hardware implementations for data flows like those found in neural networks. [13]

Facebook has released a set of extension modules as open source software. [14]

See also

Related Research Articles

Tensor Algebraic object with geometric applications

In mathematics, a tensor is an algebraic object that describes a (multilinear) relationship between sets of algebraic objects related to a vector space. Objects that tensors may map between include vectors and scalars, and even other tensors. Tensors can take several different forms – for example: scalars and vectors, dual vectors, multilinear maps between vector spaces, and even some operations such as the dot product. Tensors are defined independent of any basis, although they are often referred to by their components in a basis related to a particular coordinate system.

Nonlinear dimensionality reduction Summary of algorithms for nonlinear dimensionality reduction

High-dimensional data, meaning data that requires more than two or three dimensions to represent, can be difficult to interpret. One approach to simplification is to assume that the data of interest lies within lower-dimensional space. If the data of interest is of low enough dimension, the data can be visualised in the low-dimensional space.

In mathematics, tensor calculus, tensor analysis, or Ricci calculus is an extension of vector calculus to tensor fields.

Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable. Dimensionality reduction is common in fields that deal with large numbers of observations and/or large numbers of variables, such as signal processing, speech recognition, neuroinformatics, and bioinformatics.

Autoencoder

An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learned, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, hence its name. Variants exist, aiming to force the learned representations to assume useful properties. Examples are regularized autoencoders, which are effective in learning representations for subsequent classification tasks, and Variational autoencoders, with applications as generative models. Autoencoders are applied to many problems, from facial recognition to acquiring the semantic meaning of words.

There are many types of artificial neural networks (ANN).

Multilinear subspace learning Approach to dimensionality reduction

Multilinear subspace learning is an approach to dimensionality reduction. Dimensionality reduction can be performed on a data tensor whose observations have been vectorized and organized into a data tensor, or whose observations are matrices that are concatenated into a data tensor. Here are some examples of data tensors whose observations are vectorized or whose observations are matrices concatenated into data tensor images (2D/3D), video sequences (3D/4D), and hyperspectral cubes (3D/4D).

LuaRocks is a package manager for the Lua programming language that provides a standard format for distributing Lua modules, a tool designed to easily manage the installation of rocks, and a server for distributing them. While not included with the Lua distribution, it has been called the "de facto package manager for community-contributed Lua modules".

OpenNN is a software library written in the C++ programming language which implements neural networks, a main area of deep learning research. The library is open-source, licensed under the GNU Lesser General Public License.

A recursive neural network is a kind of deep neural network created by applying the same set of weights recursively over a structured input, to produce a structured prediction over variable-size input structures, or a scalar prediction on it, by traversing a given structure in topological order. Recursive neural networks, sometimes abbreviated as RvNNs, have been successful, for instance, in learning sequence and tree structures in natural language processing, mainly phrase and sentence continuous representations based on word embedding. RvNNs have first been introduced to learn distributed representations of structure, such as logical terms. Models and general frameworks have been developed in further works since the 1990s.

TensorFlow Machine learning software library

TensorFlow is a free and open-source software library for machine learning. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks.

The following table compares notable software frameworks, libraries and computer programs for deep learning.

Tensor Processing Unit AI accelerator ASIC by Google

Tensor Processing Unit (TPU) is an AI accelerator application-specific integrated circuit (ASIC) developed by Google specifically for neural network machine learning, particularly using Google's own TensorFlow software. Google began using TPUs internally in 2015, and in 2018 made them available for third party use, both as part of its cloud infrastructure and by offering a smaller version of the chip for sale.

spaCy

spaCy is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython. The library is published under the MIT license and its main developers are Matthew Honnibal and Ines Montani, the founders of the software company Explosion.

PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing, primarily developed by Facebook's AI Research lab (FAIR). It is free and open-source software released under the Modified BSD license. Although the Python interface is more polished and the primary focus of development, PyTorch also has a C++ interface.

In machine learning, hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters are learned.

SqueezeNet is the name of a deep neural network for computer vision that was released in 2016. SqueezeNet was developed by researchers at DeepScale, University of California, Berkeley, and Stanford University. In designing SqueezeNet, the authors' goal was to create a smaller neural network with fewer parameters that can more easily fit into computer memory and can more easily be transmitted over a computer network.

Rnn (software)

rnn is an open-source machine learning framework that implements recurrent neural network architectures, such as LSTM and GRU, natively in the R programming language, that has been downloaded over 100,000 times.

This is a comparison of statistical analysis software that allows doing inference with Gaussian processes often using approximations.

References

  1. 1 2 "Torch: a modular machine learning software library". 30 October 2002. CiteSeerX   10.1.1.8.9850 .Cite journal requires |journal= (help)
  2. Collobert, Ronan. "Torch7". GitHub .
  3. "Torch7: A Matlab-like Environment for Machine Learning" (PDF). Neural Information Processing Systems. 2011.
  4. Torch GitHub repository ReadMe
  5. PyTorch GitHub repository
  6. "Cheatsheet · torch/torch7 Wiki".
  7. KDnuggets Interview with Yann LeCun, Deep Learning Expert, Director of Facebook AI Lab
  8. Hacker News
  9. Yann Lecun's Facebook Page
  10. IDIAP Research Institute : Torch
  11. Torch-android GitHub repository
  12. Torch-ios GitHub repository
  13. NeuFlow: A Runtime Reconfigurable Dataflow Processor for Vision
  14. "Facebook Open-Sources a Trove of AI Tools". Wired . 16 January 2015.