PyTorch

Last updated
PyTorch
Original author(s)
  • Adam Paszke
  • Sam Gross
  • Soumith Chintala
  • Gregory Chanan
Developer(s) Meta AI
Initial releaseSeptember 2016;7 years ago (2016-09) [1]
Stable release
2.2.2 [2]   OOjs UI icon edit-ltr-progressive.svg / 27 March 2024;27 days ago (27 March 2024)
Repository github.com/pytorch/pytorch
Written in
Operating system
Platform IA-32, x86-64, ARM64
Available inEnglish
Type Library for machine learning and deep learning
License BSD-3 [3]
Website pytorch.org

PyTorch is a machine learning library based on the Torch library, [4] [5] [6] used for applications such as computer vision and natural language processing, [7] originally developed by Meta AI and now part of the Linux Foundation umbrella. [8] [9] [10] [11] It is recognized as one of the two most popular machine learning libraries alongside TensorFlow, offering free and open-source software released under the modified BSD license. Although the Python interface is more polished and the primary focus of development, PyTorch also has a C++ interface. [12]

Contents

A number of pieces of deep learning software are built on top of PyTorch, including Tesla Autopilot, [13] Uber's Pyro, [14] Hugging Face's Transformers, [15] PyTorch Lightning, [16] [17] and Catalyst. [18] [19]

PyTorch provides two high-level features: [20]

History

Meta (formerly known as Facebook) operates both PyTorch and Convolutional Architecture for Fast Feature Embedding (Caffe2), but models defined by the two frameworks were mutually incompatible. The Open Neural Network Exchange (ONNX) project was created by Meta and Microsoft in September 2017 for converting models between frameworks. Caffe2 was merged into PyTorch at the end of March 2018. [21] In September 2022, Meta announced that PyTorch would be governed by PyTorch Foundation, a newly created independent organization –a subsidiary of Linux Foundation. [22]

PyTorch 2.0 was released on 15 March 2023. [23]

PyTorch tensors

PyTorch defines a class called Tensor (torch.Tensor) to store and operate on homogeneous multidimensional rectangular arrays of numbers. PyTorch Tensors are similar to NumPy Arrays, but can also be operated on a CUDA-capable NVIDIA GPU. PyTorch has also been developing support for other GPU platforms, for example, AMD's ROCm [24] and Apple's Metal Framework. [25]

PyTorch supports various sub-types of Tensors. [26]

Note that the term "tensor" here does not carry the same meaning as tensor in mathematics or physics. The meaning of the word in machine learning is only superficially related to its original meaning as a certain kind of object in linear algebra. Tensors in PyTorch are simply multi-dimensional arrays.

PyTorch neural networks

PyTorch defines a module called nn (torch.nn) to describe neural networks and to support training. This module offers a comprehensive collection of building blocks for neural networks, including various layers and activation functions, enabling the construction of complex models.

Example

The following program shows the low-level functionality of the library with a simple example

importtorchdtype=torch.floatdevice=torch.device("cpu")# Execute all calculations on the CPU# device = torch.device("cuda:0")  # Executes all calculations on the GPU# Create a tensor and fill it with random numbersa=torch.randn(2,3,device=device,dtype=dtype)print(a)# Output: tensor([[-1.1884,  0.8498, -1.7129],#                  [-0.8816,  0.1944,  0.5847]])b=torch.randn(2,3,device=device,dtype=dtype)print(b)# Output: tensor([[ 0.7178, -0.8453, -1.3403],#                  [ 1.3262,  1.1512, -1.7070]])print(a*b)# Output: tensor([[-0.8530, -0.7183,  2.58],#                  [-1.1692,  0.2238, -0.9981]])print(a.sum())# Output: tensor(-2.1540)print(a[1,2])# Output of the element in the third column of the second row (zero based)# Output: tensor(0.5847)print(a.max())# Output: tensor(0.8498)

The following code-block shows an example of the higher level functionality provided nn module. A neural network with linear layers is defined in the example.

importtorchfromtorchimportnn# Import the nn sub-module from PyTorchclassNeuralNetwork(nn.Module):# Neural networks are defined as classesdef__init__(self):# Layers and variables are defined in the __init__ methodsuper().__init__()# Must be in every network.self.flatten=nn.Flatten()# Construct a flattening layer.self.linear_relu_stack=nn.Sequential(# Construct a stack of layers.nn.Linear(28*28,512),# Linear Layers have an input and output shapenn.ReLU(),# ReLU is one of many activation functions provided by nnnn.Linear(512,512),nn.ReLU(),nn.Linear(512,10),)defforward(self,x):# This function defines the forward pass.x=self.flatten(x)logits=self.linear_relu_stack(x)returnlogits

See also

Related Research Articles

<span class="mw-page-title-main">CUDA</span> Parallel computing platform and programming model

Compute Unified Device Architecture (CUDA) is a parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing units (GPUs) for accelerated general-purpose processing, an approach called general-purpose computing on GPUs (GPGPU). CUDA API and its runtime: The CUDA API is an extension of the C programming language that adds the ability to specify thread-level parallelism in C and also to specify GPU device specific operations (like moving data between the CPU and the GPU). CUDA is a software layer that gives direct access to the GPU's virtual instruction set and parallel computational elements for the execution of compute kernels. In addition to drivers and runtime kernels, the CUDA platform includes compilers, libraries and developer tools to help programmers accelerate their applications.

<span class="mw-page-title-main">Torch (machine learning)</span> Deep learning software

Torch is an open-source machine learning library, a scientific computing framework, and a scripting language based on Lua. It provides LuaJIT interfaces to deep learning algorithms implemented in C. It was created by the Idiap Research Institute at EPFL. Torch development moved in 2017 to PyTorch, a port of the library to Python.

Eclipse Deeplearning4j is a programming library written in Java for the Java virtual machine (JVM). It is a framework with wide support for deep learning algorithms. Deeplearning4j includes implementations of the restricted Boltzmann machine, deep belief net, deep autoencoder, stacked denoising autoencoder and recursive neural tensor network, word2vec, doc2vec, and GloVe. These algorithms all include distributed parallel versions that integrate with Apache Hadoop and Spark.

<span class="mw-page-title-main">TensorFlow</span> Machine learning software library

TensorFlow is a free and open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks.

The following table compares notable software frameworks, libraries and computer programs for deep learning.

<span class="mw-page-title-main">Tensor Processing Unit</span> AI accelerator ASIC by Google

Tensor Processing Unit (TPU) is an AI accelerator application-specific integrated circuit (ASIC) developed by Google for neural network machine learning, using Google's own TensorFlow software. Google began using TPUs internally in 2015, and in 2018 made them available for third-party use, both as part of its cloud infrastructure and by offering a smaller version of the chip for sale.

An AI accelerator, deep learning processor, or neural processing unit (NPU) is a class of specialized hardware accelerator or computer system designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and machine vision. Typical applications include algorithms for robotics, Internet of Things, and other data-intensive or sensor-driven tasks. They are often manycore designs and generally focus on low-precision arithmetic, novel dataflow architectures or in-memory computing capability. As of 2024, a typical AI integrated circuit chip contains tens of billions of MOSFET transistors.

<span class="mw-page-title-main">Keras</span> Neural network library

Keras is an open-source library that provides a Python interface for artificial neural networks. Keras acts as an interface for the TensorFlow library.

<span class="mw-page-title-main">AMD Instinct</span> Brand name by AMD; data center GPUs for high-performance-computing, machine learning

AMD Instinct is AMD's brand of data center GPUs. It replaced AMD's FirePro S brand in 2016. Compared to the Radeon brand of mainstream consumer/gamer products, the instinct product line is intended to accelerate deep learning, artificial neural network, and high-performance computing/GPGPU applications.

Chainer is an open source deep learning framework written purely in Python on top of NumPy and CuPy Python libraries. The development is led by Japanese venture company Preferred Networks in partnership with IBM, Intel, Microsoft, and Nvidia.

Caffe is a deep learning framework, originally developed at University of California, Berkeley. It is open source, under a BSD license. It is written in C++, with a Python interface.

The Open Neural Network Exchange (ONNX) [] is an open-source artificial intelligence ecosystem of technology companies and research organizations that establish open standards for representing machine learning algorithms and software tools to promote innovation and collaboration in the AI sector. ONNX is available on GitHub.

<span class="mw-page-title-main">ROCm</span> Parallel computing platform: GPGPU libraries and application programming interface

ROCm is an Advanced Micro Devices (AMD) software stack for graphics processing unit (GPU) programming. ROCm spans several domains: general-purpose computing on graphics processing units (GPGPU), high performance computing (HPC), heterogeneous computing. It offers several programming models: HIP, OpenMP/Message Passing Interface (MPI), and OpenCL.

<span class="mw-page-title-main">Transformer (deep learning architecture)</span> Machine learning algorithm used for natural-language processing

A transformer is a deep learning architecture developed by Google and based on the multi-head attention mechanism, proposed in a 2017 paper "Attention Is All You Need". Text is converted to numerical representations called tokens, and each token is converted into a vector via looking up from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism allowing the signal for key tokens to be amplified and less important tokens to be diminished. The transformer paper, published in 2017, is based on the softmax-based attention mechanism proposed by Bahdanau et. al. in 2014 for machine translation, and the Fast Weight Controller, similar to a transformer, proposed in 1992.

<span class="mw-page-title-main">Horovod (machine learning)</span>

Horovod is a free and open-source software framework for distributed deep learning training using TensorFlow, Keras, PyTorch, and Apache MXNet. Horovod is hosted under the Linux Foundation AI. Horovod has the goal of improving the speed, scale, and resource allocation when training a machine learning model.

<span class="mw-page-title-main">Owl Scientific Computing</span> Numerical programming library for the OCaml programming language

Owl Scientific Computing is a software system for scientific and engineering computing developed in the Department of Computer Science and Technology, University of Cambridge. The System Research Group (SRG) in the department recognises Owl as one of the representative systems developed in SRG in the 2010s. The source code is licensed under the MIT License and can be accessed from the GitHub repository.

Specialized computer hardware is often used to execute artificial intelligence (AI) programs faster, and with less energy, such as Lisp machines, neuromorphic engineering, event cameras, and physical neural networks. As of 2023, the market for AI hardware is dominated by GPUs.

CuPy is an open source library for GPU-accelerated computing with Python programming language, providing support for multi-dimensional arrays, sparse matrices, and a variety of numerical algorithms implemented on top of them. CuPy shares the same API set as NumPy and SciPy, allowing it to be a drop-in replacement to run NumPy/SciPy code on GPU. CuPy supports Nvidia CUDA GPU platform, and AMD ROCm GPU platform starting in v9.0.

<span class="mw-page-title-main">Google JAX</span> Machine Learning framework designed for parallelization and autograd.

Google JAX is a machine learning framework for transforming numerical functions, to be used in Python. It is described as bringing together a modified version of autograd and TensorFlow's XLA. It is designed to follow the structure and workflow of NumPy as closely as possible and works with various existing frameworks such as TensorFlow and PyTorch. The primary functions of JAX are:

  1. grad: automatic differentiation
  2. jit: compilation
  3. vmap: auto-vectorization
  4. pmap: SPMD programming

Tensor informally refers in machine learning to two different concepts that organize and represent data. Data may be organized in a multidimensional array (M-way array) that is informally referred to as a "data tensor"; however in the strict mathematical sense, a tensor is a multilinear mapping over a set of domain vector spaces to a range vector space. Observations, such as images, movies, volumes, sounds, and relationships among words and concepts, stored in an M-way array ("data tensor") may be analyzed either by artificial neural networks or tensor methods.

References

  1. ↑ Chintala, Soumith (1 September 2016). "PyTorch Alpha-1 release".
  2. ↑ "Release 2.2.2". 27 March 2024. Retrieved 1 April 2024.
  3. ↑ Claburn, Thomas (12 September 2022). "PyTorch gets lit under The Linux Foundation". The Register .
  4. ↑ Yegulalp, Serdar (19 January 2017). "Facebook brings GPU-powered machine learning to Python". InfoWorld. Retrieved 11 December 2017.
  5. ↑ Lorica, Ben (3 August 2017). "Why AI and machine learning researchers are beginning to embrace PyTorch". O'Reilly Media. Retrieved 11 December 2017.
  6. ↑ Ketkar, Nikhil (2017). "Introduction to PyTorch". Deep Learning with Python. Apress, Berkeley, CA. pp. 195–208. doi:10.1007/978-1-4842-2766-4_12. ISBN   9781484227657.
  7. ↑ Moez Ali (Jun 2023). "NLP with PyTorch: A Comprehensive Guide". datacamp.com. Retrieved 2024-04-01.
  8. ↑ Patel, Mo (2017-12-07). "When two trends fuse: PyTorch and recommender systems". O'Reilly Media. Retrieved 2017-12-18.
  9. ↑ Mannes, John. "Facebook and Microsoft collaborate to simplify conversions from PyTorch to Caffe2". TechCrunch . Retrieved 2017-12-18. FAIR is accustomed to working with PyTorch – a deep learning framework optimized for achieving state of the art results in research, regardless of resource constraints. Unfortunately in the real world, most of us are limited by the computational capabilities of our smartphones and computers.
  10. ↑ Arakelyan, Sophia (2017-11-29). "Tech giants are using open source frameworks to dominate the AI community". VentureBeat . Retrieved 2017-12-18.
  11. ↑ "PyTorch strengthens its governance by joining the Linux Foundation". pytorch.org. Retrieved 2022-09-13.
  12. ↑ "The C++ Frontend". PyTorch Master Documentation. Retrieved 2019-07-29.
  13. ↑ Karpathy, Andrej. "PyTorch at Tesla - Andrej Karpathy, Tesla".
  14. ↑ "Uber AI Labs Open Sources Pyro, a Deep Probabilistic Programming Language". Uber Engineering Blog. 2017-11-03. Retrieved 2017-12-18.
  15. ↑ PYTORCH-TRANSFORMERS: PyTorch implementations of popular NLP Transformers, PyTorch Hub, 2019-12-01, retrieved 2019-12-01
  16. ↑ PYTORCH-Lightning: The lightweight PyTorch wrapper for ML researchers. Scale your models. Write less boilerplate, Lightning-Team, 2020-06-18, retrieved 2020-06-18
  17. ↑ "Ecosystem Tools". pytorch.org. Retrieved 2020-06-18.
  18. ↑ GitHub - catalyst-team/catalyst: Accelerated DL & RL, Catalyst-Team, 2019-12-05, retrieved 2019-12-05
  19. ↑ "Ecosystem Tools". pytorch.org. Retrieved 2020-04-04.
  20. ↑ "PyTorch – About". pytorch.org. Archived from the original on 2018-06-15. Retrieved 2018-06-11.
  21. ↑ "Caffe2 Merges With PyTorch". 2018-04-02.
  22. ↑ Edwards, Benj (2022-09-12). "Meta spins off PyTorch Foundation to make AI framework vendor neutral". Ars Technica .
  23. ↑ "PyTorch 2.0 brings new fire to open-source machine learning". VentureBeat. 15 March 2023. Retrieved 16 March 2023.
  24. ↑ "Installing PyTorch for ROCm". rocm.docs.amd.com. 2024-02-09.
  25. ↑ "Introducing Accelerated PyTorch Training on Mac". pytorch.org. Retrieved 2022-06-04.
  26. ↑ "An Introduction to PyTorch – A Simple yet Powerful Deep Learning Library". analyticsvidhya.com. 2018-02-22. Retrieved 2018-06-11.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.