MNIST database

Last updated
Sample images from MNIST test dataset MNIST dataset example.png
Sample images from MNIST test dataset

The MNIST database (Modified National Institute of Standards and Technology database [1] ) is a large database of handwritten digits that is commonly used for training various image processing systems. [2] [3] The database is also widely used for training and testing in the field of machine learning. [4] [5] It was created by "re-mixing" the samples from NIST's original datasets. [6] The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments. [7] Furthermore, the black and white images from NIST were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels. [7]

Contents

The MNIST database contains 60,000 training images and 10,000 testing images. [8] Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset. [9] The original creators of the database keep a list of some of the methods tested on it. [7] In their original paper, they use a support-vector machine to get an error rate of 0.8%. [10]

The original MNIST dataset contains at least 4 wrong labels. [11]

History

The set of images in the MNIST database was created in 1994. Previously, NIST released two datasets: Special Database 1 (NIST Test Data I, or SD-1); and Special Database 3 (or SD-2). They were released on two CD-ROMs.

SD-1 was the test set, and it contained digits written by high school students, 58,646 images written by 500 different writers. Each image is accompanied by the identity of its writer. SD-3 was the training set, and it contained digits written by 2000 employees of the United States Census Bureau. It was much cleaner and easier to recognize than images in SD-1. [7] It was found that machine learning systems trained and validated on SD-3 suffered significant drops in performance on the test set. [12]

The original dataset from MNIST contained 128x128 binary images. Each was size-normalized to fit in a 20x20 pixel box while preserving their aspect ratio, and anti-aliased to grayscale. Then it was put into a 28x28 image by translating it until the center of mass of the pixels is in the center of the image. The details of how the downsampling proceeded was reconstructed. [13]

The training set and the testing set both originally had 60k samples, but 50k of the testing set samples were discarded. These were restored to construct the QMNIST, which has 60k images in the training set and 60k in the testing set. [14] [13]

Extended MNIST (EMNIST) is a newer dataset developed and released by NIST to be the (final) successor to MNIST. [15] [16] MNIST included images only of handwritten digits. EMNIST includes all the images from NIST Special Database 19 (SD 19), which is a large database of 814,255 handwritten uppercase and lower case letters and digits. [17] [18] The images in EMNIST were converted into the same 28x28 pixel format, by the same process, as were the MNIST images. Accordingly, tools which work with the older, smaller, MNIST dataset will likely work unmodified with EMNIST.

Fashion MNIST was created in 2017 as a more challenging replacement for MNIST. The dataset consists of 70,000 28x28 grayscale images of fashion products from 10 categories. [19]

Performance

Some researchers have achieved "near-human performance" on the MNIST database, using a committee of neural networks; in the same paper, the authors achieve performance double that of humans on other recognition tasks. [20] The highest error rate listed [7] on the original website of the database is 12 percent, which is achieved using a simple linear classifier with no preprocessing. [10]

In 2004, a best-case error rate of 0.42 percent was achieved on the database by researchers using a new classifier called the LIRA, which is a neural classifier with three neuron layers based on Rosenblatt's perceptron principles. [21]

Some researchers have tested artificial intelligence systems using the database put under random distortions. The systems in these cases are usually neural networks and the distortions used tend to be either affine distortions or elastic distortions. [7] Sometimes, these systems can be very successful; one such system achieved an error rate on the database of 0.39 percent. [22]

In 2011, an error rate of 0.27 percent, improving on the previous best result, was reported by researchers using a similar system of neural networks. [23] In 2013, an approach based on regularization of neural networks using DropConnect has been claimed to achieve a 0.21 percent error rate. [24] In 2016, the single convolutional neural network best performance was 0.25 percent error rate. [25] As of August 2018, the best performance of a single convolutional neural network trained on MNIST training data using no data augmentation is 0.25 percent error rate. [25] [26] Also, the Parallel Computing Center (Khmelnytskyi, Ukraine) obtained an ensemble of only 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate. [27] [28]

Classifiers

This is a table of some of the machine learning methods used on the dataset and their error rates, by type of classifier:

TypeClassifierDistortion Preprocessing Error rate (%)
Linear classifier Pairwise linear classifier NoneDeskewing7.6 [10]
K-Nearest Neighbors K-NN with rigid transformationsNoneNone0.96 [29]
K-Nearest Neighbors K-NN with non-linear deformation (P2DHMDM)NoneShiftable edges0.52 [30]
Boosted Stumps Product of stumps on Haar features NoneHaar features0.87 [31]
Non-linear classifier40 PCA + quadratic classifierNoneNone3.3 [10]
Random Forest Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC) [32] NoneSimple statistical pixel importance2.8 [33]
Support-vector machine (SVM)Virtual SVM, deg-9 poly, 2-pixel jitteredNoneDeskewing0.56 [34]
Neural network 2-layer 784-800-10NoneNone1.6 [35]
Neural network 2-layer 784-800-10Elastic distortionsNone0.7 [35]
Deep neural network (DNN)6-layer 784-2500-2000-1500-1000-500-10Elastic distortionsNone0.35 [36]
Convolutional neural network (CNN)6-layer 784-40-80-500-1000-2000-10NoneExpansion of the training data0.31 [37]
Convolutional neural network 6-layer 784-50-100-500-1000-10-10NoneExpansion of the training data0.27 [38]
Convolutional neural network (CNN)13-layer 64-128(5x)-256(3x)-512-2048-256-256-10NoneNone0.25 [25]
Convolutional neural network Committee of 35 CNNs, 1-20-P-40-P-150-10Elastic distortionsWidth normalizations0.23 [20]
Convolutional neural network Committee of 5 CNNs, 6-layer 784-50-100-500-1000-10-10NoneExpansion of the training data0.21 [27] [28]
Convolutional neural network Committee of 20 CNNS with Squeeze-and-Excitation Networks [39] None Data augmentation 0.17 [40]
Convolutional neural network Ensemble of 3 CNNs with varying kernel sizesNone Data augmentation consisting of rotation and translation0.09 [41]

See also

Related Research Articles

<span class="mw-page-title-main">Neural network (machine learning)</span> Computational model used in machine learning, based on connected, hierarchical functions

In machine learning, a neural network is a model inspired by the structure and function of biological neural networks in animal brains.

<span class="mw-page-title-main">Handwriting recognition</span> Ability of a computer to receive and interpret intelligible handwritten input

Handwriting recognition (HWR), also known as handwritten text recognition (HTR), is the ability of a computer to receive and interpret intelligible handwritten input from sources such as paper documents, photographs, touch-screens and other devices. The image of the written text may be sensed "off line" from a piece of paper by optical scanning or intelligent word recognition. Alternatively, the movements of the pen tip may be sensed "on line", for example by a pen-based computer screen surface, a generally easier task as there are more clues available. A handwriting recognition system handles formatting, performs correct segmentation into characters, and finds the most possible words.

<span class="mw-page-title-main">Jürgen Schmidhuber</span> German computer scientist

Jürgen Schmidhuber is a German computer scientist noted for his work in the field of artificial intelligence, specifically artificial neural networks. He is a scientific director of the Dalle Molle Institute for Artificial Intelligence Research in Switzerland. He is also director of the Artificial Intelligence Initiative and professor of the Computer Science program in the Computer, Electrical, and Mathematical Sciences and Engineering (CEMSE) division at the King Abdullah University of Science and Technology (KAUST) in Saudi Arabia.

<span class="mw-page-title-main">Optical neural network</span> Physical implementation of an artificial neural network with optical components

An optical neural network is a physical implementation of an artificial neural network with optical components. Early optical neural networks used a photorefractive Volume hologram to interconnect arrays of input neurons to arrays of output with synaptic weights in proportion to the multiplexed hologram's strength. Volume holograms were further multiplexed using spectral hole burning to add one dimension of wavelength to space to achieve four dimensional interconnects of two dimensional arrays of neural inputs and outputs. This research led to extensive research on alternative methods using the strength of the optical interconnect for implementing neuronal communications.

<span class="mw-page-title-main">Object detection</span> Computer technology related to computer vision and image processing

Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class in digital images and videos. Well-researched domains of object detection include face detection and pedestrian detection. Object detection has applications in many areas of computer vision, including image retrieval and video surveillance.

<span class="mw-page-title-main">Yann LeCun</span> French computer scientist (born 1960)

Yann André LeCun is a French-American computer scientist working primarily in the fields of machine learning, computer vision, mobile robotics and computational neuroscience. He is the Silver Professor of the Courant Institute of Mathematical Sciences at New York University and Vice President, Chief AI Scientist at Meta.

<span class="mw-page-title-main">Deep learning</span> Branch of machine learning

Deep learning is a subset of machine learning that focuses on utilizing neural networks to perform tasks such as classification, regression, and representation learning. The field takes inspiration from biological neuroscience and is centered around stacking artificial neurons into layers and "training" them to process data. The adjective "deep" refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

A convolutional neural network (CNN) is a regularized type of feed-forward neural network that learns features by itself via filter optimization. This type of deep learning network has been applied to process and make predictions from many different types of data including text, images and audio. Convolution-based networks are the de-facto standard in deep learning-based approaches to computer vision and image processing, and have only recently have been replaced -- in some cases -- by newer deep learning architectures such as the transformer. Vanishing gradients and exploding gradients, seen during backpropagation in earlier neural networks, are prevented by using regularized weights over fewer connections. For example, for each neuron in the fully-connected layer, 10,000 weights would be required for processing an image sized 100 × 100 pixels. However, applying cascaded convolution kernels, only 25 neurons are required to process 5x5-sized tiles. Higher-layer features are extracted from wider context windows, compared to lower-layer features.

This page is a timeline of machine learning. Major discoveries, achievements, milestones and other major events in machine learning are included.

The ImageNet project is a large visual database designed for use in visual object recognition software research. More than 14 million images have been hand-annotated by the project to indicate what objects are pictured and in at least one million of the images, bounding boxes are also provided. ImageNet contains more than 20,000 categories, with a typical category, such as "balloon" or "strawberry", consisting of several hundred images. The database of annotations of third-party image URLs is freely available directly from ImageNet, though the actual images are not owned by ImageNet. Since 2010, the ImageNet project runs an annual software contest, the ImageNet Large Scale Visual Recognition Challenge, where software programs compete to correctly classify and detect objects and scenes. The challenge uses a "trimmed" list of one thousand non-overlapping classes.

Data augmentation is a statistical technique which allows maximum likelihood estimation from incomplete data. Data augmentation has important applications in Bayesian analysis, and the technique is widely used in machine learning to reduce overfitting when training machine learning models, achieved by training models on several slightly-modified copies of existing data.

<span class="mw-page-title-main">AlexNet</span> An influential convolutional neural network published in 2012

AlexNet is a convolutional neural network (CNN) architecture, designed by Alex Krizhevsky in collaboration with Ilya Sutskever and Geoffrey Hinton, who was Krizhevsky's Ph.D. advisor at the University of Toronto in 2012. It had 60 million parameters and 650,000 neurons.

The CIFAR-10 dataset is a collection of images that are commonly used to train machine learning and computer vision algorithms. It is one of the most widely used datasets for machine learning research. The CIFAR-10 dataset contains 60,000 32x32 color images in 10 different classes. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each class.

Neural architecture search (NAS) is a technique for automating the design of artificial neural networks (ANN), a widely used model in the field of machine learning. NAS has been used to design networks that are on par with or outperform hand-designed architectures. Methods for NAS can be categorized according to the search space, search strategy and performance estimation strategy used:

In artificial neural networks, a convolutional layer is a type of network layer that applies a convolution operation to the input. Convolutional layers are some of the primary building blocks of convolutional neural networks (CNNs), a class of neural network most commonly applied to images, video, audio, and other data that have the property of uniform translational symmetry.

Artificial neural networks (ANNs) are models created using machine learning to perform a number of tasks. Their creation was inspired by biological neural circuitry. While some of the computational implementations ANNs relate to earlier discoveries in mathematics, the first implementation of ANNs was by psychologist Frank Rosenblatt, who developed the perceptron. Little research was conducted on ANNs in the 1970s and 1980s, with the AAAI calling this period an "AI winter".

<span class="mw-page-title-main">LeNet</span> Convolutional neural network structure

LeNet is a series of convolutional neural network structure proposed by LeCun et al.. The earliest version, LeNet-1, was trained in 1989. In general, when "LeNet" is referred to without a number, it refers to LeNet-5 (1998), the most well-known version.

The Fashion MNIST dataset is a large freely available database of fashion images that is commonly used for training and testing various machine learning systems. Fashion-MNIST was intended to serve as a replacement for the original MNIST database for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits.

Clément Farabet is a computer scientist and AI expert known for his contributions to the field of deep learning. He served as a research scientist at the New York University. He serves as the Vice President of Research at Google DeepMind and previously served as the VP of AI Infrastructure at NVIDIA.

References

  1. ↑ "The MNIST Database of handwritten digits". Yann LeCun, Courant Institute, NYU Corinna Cortes, Google Labs, New York Christopher J.C. Burges, Microsoft Research, Redmond.
  2. ↑ "Support vector machines speed pattern recognition - Vision Systems Design". Vision Systems Design. September 2004. Retrieved 17 August 2013.
  3. ↑ Gangaputra, Sachin. "Handwritten digit database" . Retrieved 17 August 2013.
  4. ↑ Qiao, Yu (2007). "The MNIST Database of handwritten digits" . Retrieved 18 August 2013.
  5. ↑ Platt, John C. (1999). "Using analytic QP and sparseness to speed training of support vector machines" (PDF). Advances in Neural Information Processing Systems: 557–563. Archived from the original (PDF) on 4 March 2016. Retrieved 18 August 2013.
  6. ↑ Grother, Patrick J. "NIST Special Database 19 - Handprinted Forms and Characters Database" (PDF). National Institute of Standards and Technology .
  7. 1 2 3 4 5 6 LeCun, Yann; Cortez, Corinna; Burges, Christopher C.J. "The MNIST Handwritten Digit Database". Yann LeCun's Website yann.lecun.com. Retrieved 30 April 2020.
  8. ↑ Kussul, Ernst; Baidyk, Tatiana (2004). "Improved method of handwritten digit recognition tested on MNIST database". Image and Vision Computing. 22 (12): 971–981. doi:10.1016/j.imavis.2004.03.008.
  9. ↑ Zhang, Bin; Srihari, Sargur N. (2004). "Fast k-Nearest Neighbor Classification Using Cluster-Based Trees" (PDF). IEEE Transactions on Pattern Analysis and Machine Intelligence. 26 (4): 525–528. doi:10.1109/TPAMI.2004.1265868. PMID   15382657. S2CID   6883417 . Retrieved 20 April 2020.
  10. 1 2 3 4 LeCun, Yann; Léon Bottou; Yoshua Bengio; Patrick Haffner (1998). "Gradient-Based Learning Applied to Document Recognition" (PDF). Proceedings of the IEEE. 86 (11): 2278–2324. doi:10.1109/5.726791. S2CID   14542261 . Retrieved 18 August 2013.
  11. ↑ Muller, Nicolas M.; Markert, Karla (July 2019). Identifying Mislabeled Instances in Classification Datasets. 2019 International Joint Conference on Neural Networks (IJCNN). IEEE. pp. 1–8. arXiv: 1912.05283 . doi:10.1109/IJCNN.2019.8851920. ISBN   978-1-7281-1985-4.
  12. ↑ Bottou, Léon; Cortes, Corinna; Denker, John S.; Drucker, Harris; Guyon, Isabelle; Jackel, L. D.; LeCun, Y.; Muller, U. A.; Sackinger, E.; Simard, P.; Vapnik, V. (1994). "Comparison of classifier methods: A case study in handwritten digit recognition". Proceedings of the 12th IAPR International Conference on Pattern Recognition (Cat. No.94CH3440-5). Vol. 2. Jerusalem, Israel. pp. 77–82. doi:10.1109/ICPR.1994.576879. ISBN   0-8186-6270-0.{{cite book}}: CS1 maint: location missing publisher (link)
  13. 1 2 Yadav, Chhavi; Bottou, Leon (2019). "Cold Case: The Lost MNIST Digits". Advances in Neural Information Processing Systems. 32. arXiv: 1905.10498 . Article has a detailed history and a reconstruction of the discarded testing set.
  14. ↑ facebookresearch/qmnist, Meta Research, 2024-09-23, retrieved 2024-10-25
  15. ↑ NIST (4 April 2017). "The EMNIST Dataset". NIST . Retrieved 11 April 2022.
  16. ↑ NIST (27 August 2010). "NIST Special Database 19". NIST . Retrieved 11 April 2022.
  17. ↑ Cohen, G.; Afshar, S.; Tapson, J.; van Schaik, A. (2017). "EMNIST: an extension of MNIST to handwritten letters". arXiv: 1702.05373 [cs.CV].
  18. ↑ Grother, Patrick J., and K. K. Hanaoka. "NIST special database 19." Handprinted forms and characters database, National Institute of Standards and Technology 10 (1995): 69.
  19. ↑ Xiao, Han; Rasul, Kashif; Vollgraf, Roland (2017-09-15). "Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms". arXiv: 1708.07747 [cs.LG].
  20. 1 2 Cires¸an, Dan; Ueli Meier; Jürgen Schmidhuber (2012). "Multi-column deep neural networks for image classification" (PDF). 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3642–3649. arXiv: 1202.2745 . CiteSeerX   10.1.1.300.3283 . doi:10.1109/CVPR.2012.6248110. ISBN   978-1-4673-1228-8. S2CID   2161592.
  21. ↑ Kussul, Ernst; Tatiana Baidyk (2004). "Improved method of handwritten digit recognition tested on MNIST database" (PDF). Image and Vision Computing. 22 (12): 971–981. doi:10.1016/j.imavis.2004.03.008. Archived from the original (PDF) on 21 September 2013. Retrieved 20 September 2013.
  22. ↑ Ranzato, Marc'Aurelio; Christopher Poultney; Sumit Chopra; Yann LeCun (2006). "Efficient Learning of Sparse Representations with an Energy-Based Model" (PDF). Advances in Neural Information Processing Systems. 19: 1137–1144. Retrieved 20 September 2013.
  23. ↑ Ciresan, Dan Claudiu; Ueli Meier; Luca Maria Gambardella; Jürgen Schmidhuber (2011). "Convolutional neural network committees for handwritten character classification" (PDF). 2011 International Conference on Document Analysis and Recognition (ICDAR). pp. 1135–1139. CiteSeerX   10.1.1.465.2138 . doi:10.1109/ICDAR.2011.229. ISBN   978-1-4577-1350-7. S2CID   10122297. Archived from the original (PDF) on 22 February 2016. Retrieved 20 September 2013.
  24. ↑ Wan, Li; Matthew Zeiler; Sixin Zhang; Yann LeCun; Rob Fergus (2013). Regularization of Neural Network using DropConnect. International Conference on Machine Learning(ICML).
  25. 1 2 3 SimpleNet (2016). "Lets Keep it simple, Using simple architectures to outperform deeper and more complex architectures". arXiv: 1608.06037 . Retrieved 3 December 2020.
  26. ↑ SimpNet (2018). "Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet". Github. arXiv: 1802.06205 . Retrieved 3 December 2020.
  27. 1 2 Romanuke, Vadim. "Parallel Computing Center (Khmelnytskyi, Ukraine) represents an ensemble of 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate" . Retrieved 24 November 2016.
  28. 1 2 Romanuke, Vadim (2016). "Training data expansion and boosting of convolutional neural networks for reducing the MNIST dataset error rate". Research Bulletin of NTUU "Kyiv Polytechnic Institute". 6 (6): 29–34. doi: 10.20535/1810-0546.2016.6.84115 .
  29. ↑ Lindblad, Joakim; Nataša Sladoje (January 2014). "Linear time distances between fuzzy sets with applications to pattern matching and classification". IEEE Transactions on Image Processing. 23 (1): 126–136. Bibcode:2014ITIP...23..126L. doi:10.1109/TIP.2013.2286904. PMID   24158476. S2CID   1908950.
  30. ↑ Keysers, Daniel; Thomas Deselaers; Christian Gollan; Hermann Ney (August 2007). "Deformation models for image recognition". IEEE Transactions on Pattern Analysis and Machine Intelligence. 29 (8): 1422–1435. CiteSeerX   10.1.1.106.3963 . doi:10.1109/TPAMI.2007.1153. PMID   17568145. S2CID   2528485.
  31. ↑ Kégl, Balázs; Róbert Busa-Fekete (2009). "Boosting products of base classifiers" (PDF). Proceedings of the 26th Annual International Conference on Machine Learning. pp. 497–504. doi:10.1145/1553374.1553439. ISBN   9781605585161. S2CID   8460779 . Retrieved 27 August 2013.
  32. ↑ "RandomForestSRC: Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC)". 21 January 2020.
  33. ↑ "Mehrad Mahmoudian / MNIST with RandomForest".
  34. ↑ Decoste, Dennis; Schölkopf, Bernhard (2002). "Training Invariant Support Vector Machines". Machine Learning. 46 (1–3): 161–190. doi: 10.1023/A:1012454411458 . ISSN   0885-6125. OCLC   703649027.
  35. 1 2 Patrice Y. Simard; Dave Steinkraus; John C. Platt (2003). "Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis". Proceedings of the Seventh International Conference on Document Analysis and Recognition. Vol. 1. Institute of Electrical and Electronics Engineers. p. 958. doi:10.1109/ICDAR.2003.1227801. ISBN   978-0-7695-1960-9. S2CID   4659176.
  36. ↑ Ciresan, Claudiu Dan; Ueli Meier; Luca Maria Gambardella; Juergen Schmidhuber (December 2010). "Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition". Neural Computation. 22 (12): 3207–20. arXiv: 1003.0358 . doi:10.1162/NECO_a_00052. PMID   20858131. S2CID   1918673.
  37. ↑ Romanuke, Vadim. "The single convolutional neural network best performance in 18 epochs on the expanded training data at Parallel Computing Center, Khmelnytskyi, Ukraine" . Retrieved 16 November 2016.
  38. ↑ Romanuke, Vadim. "Parallel Computing Center (Khmelnytskyi, Ukraine) gives a single convolutional neural network performing on MNIST at 0.27 percent error rate" . Retrieved 24 November 2016.
  39. ↑ Hu, Jie; Shen, Li; Albanie, Samuel; Sun, Gang; Wu, Enhua (2019). "Squeeze-and-Excitation Networks". IEEE Transactions on Pattern Analysis and Machine Intelligence. 42 (8): 2011–2023. arXiv: 1709.01507 . doi:10.1109/TPAMI.2019.2913372. PMID   31034408. S2CID   140309863.
  40. ↑ "GitHub - Matuzas77/MNIST-0.17: MNIST classifier with average 0.17% error". GitHub . 25 February 2020.
  41. ↑ An, Sanghyeon; Lee, Minjun; Park, Sanglee; Yang, Heerin; So, Jungmin (2020-10-04). "An Ensemble of Simple Convolutional Neural Network Models for MNIST Digit Recognition". arXiv: 2008.10400 [cs.CV].

Further reading

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.