One-hot

Last updated

Decimal Binary Unary One-hot
00000000000000000001
10010000000100000010
20100000001100000100
30110000011100001000
41000000111100010000
51010001111100100000
61100011111101000000
71110111111110000000

In digital circuits and machine learning, a one-hot is a group of bits among which the legal combinations of values are only those with a single high (1) bit and all the others low (0). [1] A similar implementation in which all bits are '1' except one '0' is sometimes called one-cold. [2] In statistics, dummy variables represent a similar technique for representing categorical data.

Contents

Applications

Digital circuitry

One-hot encoding is often used for indicating the state of a state machine. When using binary, a decoder is needed to determine the state. A one-hot state machine, however, does not need a decoder as the state machine is in the nth state if, and only if, the nth bit is high.

A ring counter with 15 sequentially ordered states is an example of a state machine. A 'one-hot' implementation would have 15 flip flops chained in series with the Q output of each flip flop connected to the D input of the next and the D input of the first flip flop connected to the Q output of the 15th flip flop. The first flip flop in the chain represents the first state, the second represents the second state, and so on to the 15th flip flop, which represents the last state. Upon reset of the state machine all of the flip flops are reset to '0' except the first in the chain, which is set to '1'. The next clock edge arriving at the flip flops advances the one 'hot' bit to the second flip flop. The 'hot' bit advances in this way until the 15th state, after which the state machine returns to the first state.

An address decoder converts from binary to one-hot representation. A priority encoder converts from one-hot representation to binary.

Comparison with other encoding methods

Advantages
  • Determining the state has a low and constant cost of accessing one flip-flop
  • Changing the state has the constant cost of accessing two flip-flops
  • Easy to design and modify
  • Easy to detect illegal states
  • Takes advantage of an FPGA's abundant flip-flops
  • Using a one-hot implementation typically allows a state machine to run at a faster clock rate than any other encoding of that state machine [3]
Disadvantages
  • Requires more flip-flops than other encodings, making it impractical for PAL devices
  • Many of the states are illegal [4]

Natural language processing

In natural language processing, a one-hot vector is a 1 × N matrix (vector) used to distinguish each word in a vocabulary from every other word in the vocabulary. [5] The vector consists of 0s in all cells with the exception of a single 1 in a cell used uniquely to identify the word. One-hot encoding ensures that machine learning does not assume that higher numbers are more important. For example, the value '8' is bigger than the value '1', but that does not make '8' more important than '1'. The same is true for words: the value 'laughter' is not more important than 'laugh'.

Machine learning and statistics

In machine learning, one-hot encoding is a frequently used method to deal with categorical data. Because many machine learning models need their input variables to be numeric, categorical variables need to be transformed in the pre-processing part. [6]

Label Encoding
Food NameCategorical #Calories
Apple195
Chicken2231
Broccoli350
One Hot Encoding
AppleChickenBroccoliCalories
10095
010231
00150

Categorical data can be either nominal or ordinal. [7] Ordinal data has a ranked order for its values and can therefore be converted to numerical data through ordinal encoding. [8] An example of ordinal data would be the ratings on a test ranging from A to F, which could be ranked using numbers from 6 to 1. Since there is no quantitative relationship between nominal variables' individual values, using ordinal encoding can potentially create a fictional ordinal relationship in the data. [9] Therefore, one-hot encoding is often applied to nominal variables, in order to improve the performance of the algorithm.

For each unique value in the original categorical column, a new column is created in this method. These dummy variables are then filled up with zeros and ones (1 meaning TRUE, 0 meaning FALSE).[ citation needed ]

Because this process creates multiple new variables, it is prone to creating a 'big p' problem (too many predictors) if there are many unique values in the original column. Another downside of one-hot encoding is that it causes multicollinearity between the individual variables, which potentially reduces the model's accuracy.[ citation needed ]

Also, if the categorical variable is an output variable, you may want to convert the values back into a categorical form in order to present them in your application. [10]

In practical usage, this transformation is often directly performed by a function that takes categorical data as an input and outputs the corresponding dummy variables. An example would be the dummyVars function of the Caret library in R. [11]

See also

Related Research Articles

The bit is the most basic unit of information in computing and digital communications. The name is a portmanteau of binary digit. The bit represents a logical state with one of two possible values. These values are most commonly represented as either "1" or "0", but other representations such as true/false, yes/no, on/off, or +/− are also widely used.

In digital logic and computing, a counter is a device which stores the number of times a particular event or process has occurred, often in relationship to a clock. The most common type is a sequential digital logic circuit with an input line called the clock and multiple output lines. The values on the output lines represent a number in the binary or BCD number system. Each pulse applied to the clock input increments or decrements the number in the counter.

A shift register is a type of digital circuit using a cascade of flip-flops where the output of one flip-flop is connected to the input of the next. They share a single clock signal, which causes the data stored in the system to shift from one location to the next. By connecting the last flip-flop back to the first, the data can cycle within the shifters for extended periods, and in this configuration they were used as computer memory, displacing delay-line memory systems in the late 1960s and early 1970s.

Verilog, standardized as IEEE 1364, is a hardware description language (HDL) used to model electronic systems. It is most commonly used in the design and verification of digital circuits at the register-transfer level of abstraction. It is also used in the verification of analog circuits and mixed-signal circuits, as well as in the design of genetic circuits. In 2009, the Verilog standard was merged into the SystemVerilog standard, creating IEEE Standard 1800-2009. Since then, Verilog has been officially part of the SystemVerilog language. The current version is IEEE standard 1800-2023.

Lempel–Ziv–Welch (LZW) is a universal lossless data compression algorithm created by Abraham Lempel, Jacob Ziv, and Terry Welch. It was published by Welch in 1984 as an improved implementation of the LZ78 algorithm published by Lempel and Ziv in 1978. The algorithm is simple to implement and has the potential for very high throughput in hardware implementations. It is the algorithm of the Unix file compression utility compress and is used in the GIF image format.

In computer programming, Base64 is a group of binary-to-text encoding schemes that transforms binary data into a sequence of printable characters, limited to a set of 64 unique characters. More specifically, the source binary data is taken 6 bits at a time, then this group of 6 bits is mapped to one of 64 unique characters.

In regression analysis, a dummy variable is one that takes a binary value to indicate the absence or presence of some categorical effect that may be expected to shift the outcome. For example, if we were studying the relationship between biological sex and income, we could use a dummy variable to represent the sex of each individual in the study. The variable could take on a value of 1 for males and 0 for females. In machine learning this is known as one-hot encoding.

In digital electronics, a binary decoder is a combinational logic circuit that converts binary information from the n coded inputs to a maximum of 2n unique outputs. They are used in a wide variety of applications, including instruction decoding, data multiplexing and data demultiplexing, seven segment displays, and as address decoders for memory and port-mapped I/O.

The Lempel–Ziv–Markov chain algorithm (LZMA) is an algorithm used to perform lossless data compression. It has been under development since either 1996 or 1998 by Igor Pavlov and was first used in the 7z format of the 7-Zip archiver. This algorithm uses a dictionary compression scheme somewhat similar to the LZ77 algorithm published by Abraham Lempel and Jacob Ziv in 1977 and features a high compression ratio and a variable compression-dictionary size, while still maintaining decompression speed similar to other commonly used compression algorithms.

In statistics, a categorical variable is a variable that can take on one of a limited, and usually fixed, number of possible values, assigning each individual or other unit of observation to a particular group or nominal category on the basis of some qualitative property. In computer science and some branches of mathematics, categorical variables are referred to as enumerations or enumerated types. Commonly, each of the possible values of a categorical variable is referred to as a level. The probability distribution associated with a random categorical variable is called a categorical distribution.

In machine learning and pattern recognition, a feature is an individual measurable property or characteristic of a phenomenon. Choosing informative, discriminating and independent features is a crucial element of effective algorithms in pattern recognition, classification and regression. Features are usually numeric, but structural features such as strings and graphs are used in syntactic pattern recognition. The concept of "feature" is related to that of explanatory variable used in statistical techniques such as linear regression.

Binary data is data whose unit can take on only two possible states. These are often labelled as 0 and 1 in accordance with the binary numeral system and Boolean algebra.

In statistics, classification is the problem of identifying which of a set of categories (sub-populations) an observation belongs to. Examples are assigning a given email to the "spam" or "non-spam" class, and assigning a diagnosis to a given patient based on observed characteristics of the patient.

A ring counter is a type of counter composed of flip-flops connected into a shift register, with the output of the last flip-flop fed to the input of the first, making a "circular" or "ring" structure.

XOR gate is a digital logic gate that gives a true output when the number of true inputs is odd. An XOR gate implements an exclusive or from mathematical logic; that is, a true output results if one, and only one, of the inputs to the gate is true. If both inputs are false (0/LOW) or both are true, a false output results. XOR represents the inequality function, i.e., the output is true if the inputs are not alike otherwise the output is false. A way to remember XOR is "must have one or the other but not both".

LEB128 or Little Endian Base 128 is a variable-length code compression used to store arbitrarily large integers in a small number of bytes. LEB128 is used in the DWARF debug file format and the WebAssembly binary encoding for all integer literals.

<span class="mw-page-title-main">Priority encoder</span> Digital electronic circuit

A priority encoder is a circuit or algorithm that compresses multiple binary inputs into a smaller number of outputs, similar to a simple encoder. The output of a priority encoder is the binary representation of the index of the most significant activated line. In contrast to the simple encoder, if two or more inputs to the priority encoder are active at the same time, the input having the highest priority will take precedence. It is an improvement on a simple encoder because it can handle all possible input combinations, but at the cost of extra logic.

<span class="mw-page-title-main">Flip-flop (electronics)</span> Electronic circuit with two stable states

In electronics, flip-flops and latches are circuits that have two stable states that can store state information – a bistable multivibrator. The circuit can be made to change state by signals applied to one or more control inputs and will output its state. It is the basic storage element in sequential logic. Flip-flops and latches are fundamental building blocks of digital electronics systems used in computers, communications, and many other types of systems.

State encoding assigns a unique pattern of ones and zeros to each defined state of a finite-state machine (FSM). Traditionally, design criteria for FSM synthesis were speed, area or both. Following Moore's law, with technology advancement, density and speed of integrated circuits have increased exponentially. With this, power dissipation per area has inevitably increased, which has forced designers for portable computing devices and high-speed processors to consider power dissipation as a critical parameter during design consideration.

Seq2seq is a family of machine learning approaches used for natural language processing. Applications include language translation, image captioning, conversational models, and text summarization. Seq2seq uses sequence transformation: it turns one sequence into another sequence.

References

  1. ↑ Harris, David and Harris, Sarah (2012-08-07). Digital design and computer architecture (2nd ed.). San Francisco, Calif.: Morgan Kaufmann. p. 129. ISBN   978-0-12-394424-5.{{cite book}}: CS1 maint: multiple names: authors list (link)
  2. ↑ Harrag, Fouzi; Gueliani, Selmene (2020). "Event Extraction Based on Deep Learning in Food Hazard Arabic Texts". arXiv: 2008.05014 .{{cite journal}}: Cite journal requires |journal= (help)
  3. ↑ Xilinx. "HDL Synthesis for FPGAs Design Guide". section 3.13: "Encoding State Machines". Appendix A: "Accelerate FPGA Macros with One-Hot Approach". 1995.
  4. ↑ Cohen, Ben (2002). Real Chip Design and Verification Using Verilog and VHDL. Palos Verdes Peninsula, CA, US: VhdlCohen Publishing. p. 48. ISBN   0-9705394-2-8.
  5. ↑ Arnaud, Émilien; Elbattah, Mahmoud; Gignon, Maxime; Dequen, Gilles (August 2021). NLP-Based Prediction of Medical Specialties at Hospital Admission Using Triage Notes. 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI). Victoria, British Columbia. pp. 548–553. doi:10.1109/ICHI52183.2021.00103 . Retrieved 2022-05-22.
  6. ↑ Brownlee, Jason. (2017). "Why One-Hot Encode Data in Machine Learning?". Machinelearningmastery. https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/
  7. ↑ Stevens, S. S. (1946). “On the Theory of Scales of Measurement”. Science, New Series, 103.2684, pp. 677–680. http://www.jstor.org/stable/1671815.
  8. ↑ Brownlee, Jason. (2020). "Ordinal and One-Hot Encodings for Categorical Data". Machinelearningmastery. https://machinelearningmastery.com/one-hot-encoding-for-categorical-data//
  9. ↑ Brownlee, Jason. (2020). "Ordinal and One-Hot Encodings for Categorical Data". Machinelearningmastery. https://machinelearningmastery.com/one-hot-encoding-for-categorical-data//
  10. ↑ Brownlee, Jason. (2017). "Why One-Hot Encode Data in Machine Learning?". Machinelearningmastery. https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/
  11. ↑ Kuhn, Max. “dummyVars”. RDocumentation. https://www.rdocumentation.org/packages/caret/versions/6.0-86/topics/dummyVars
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.