IEEE 754-1985

Last updated

IEEE 754-1985 [1] is a historic industry standard for representing floating-point numbers in computers, officially adopted in 1985 and superseded in 2008 by IEEE 754-2008, and then again in 2019 by minor revision IEEE 754-2019. [2] During its 23 years, it was the most widely used format for floating-point computation. It was implemented in software, in the form of floating-point libraries, and in hardware, in the instructions of many CPUs and FPUs. The first integrated circuit to implement the draft of what was to become IEEE 754-1985 was the Intel 8087.

Contents

IEEE 754-1985 represents numbers in binary, providing definitions for four levels of precision, of which the two most commonly used are:

LevelWidthRange at full precision Precision [lower-alpha 1]
Single precision32 bits±1.18×10−38 to ±3.4×1038Approximately 7 decimal digits
Double precision64 bits±2.23×10−308 to ±1.80×10308Approximately 16 decimal digits

The standard also defines representations for positive and negative infinity, a "negative zero", five exceptions to handle invalid results like division by zero, special values called NaNs for representing those exceptions, denormal numbers to represent numbers smaller than shown above, and four rounding modes.

Representation of numbers

The number 0.15625 represented as a single-precision IEEE 754-1985 floating-point number. See text for explanation. IEEE 754 Single Floating Point Format.svg
The number 0.15625 represented as a single-precision IEEE 754-1985 floating-point number. See text for explanation.
The three fields in a 64bit IEEE 754 float IEEE 754 Double Floating Point Format.svg
The three fields in a 64bit IEEE 754 float

Floating-point numbers in IEEE 754 format consist of three fields: a sign bit, a biased exponent, and a fraction. The following example illustrates the meaning of each.

The decimal number 0.1562510 represented in binary is 0.001012 (that is, 1/8 + 1/32). (Subscripts indicate the number base.) Analogous to scientific notation, where numbers are written to have a single non-zero digit to the left of the decimal point, we rewrite this number so it has a single 1 bit to the left of the "binary point". We simply multiply by the appropriate power of 2 to compensate for shifting the bits left by three positions:

Now we can read off the fraction and the exponent: the fraction is .012 and the exponent is −3.

As illustrated in the pictures, the three fields in the IEEE 754 representation of this number are:

sign = 0, because the number is positive. (1 indicates negative.)
biased exponent = −3 + the "bias". In single precision, the bias is 127, so in this example the biased exponent is 124; in double precision, the bias is 1023, so the biased exponent in this example is 1020.
fraction = .01000…2.

IEEE 754 adds a bias to the exponent so that numbers can in many cases be compared conveniently by the same hardware that compares signed 2's-complement integers. Using a biased exponent, the lesser of two positive floating-point numbers will come out "less than" the greater following the same ordering as for sign and magnitude integers. If two floating-point numbers have different signs, the sign-and-magnitude comparison also works with biased exponents. However, if both biased-exponent floating-point numbers are negative, then the ordering must be reversed. If the exponent were represented as, say, a 2's-complement number, comparison to see which of two numbers is greater would not be as convenient.

The leading 1 bit is omitted since all numbers except zero start with a leading 1; the leading 1 is implicit and doesn't actually need to be stored which gives an extra bit of precision for "free."

Zero

The number zero is represented specially:

sign = 0 for positive zero, 1 for negative zero.
biased exponent = 0.
fraction = 0.

Denormalized numbers

The number representations described above are called normalized, meaning that the implicit leading binary digit is a 1. To reduce the loss of precision when an underflow occurs, IEEE 754 includes the ability to represent fractions smaller than are possible in the normalized representation, by making the implicit leading digit a 0. Such numbers are called denormal. They don't include as many significant digits as a normalized number, but they enable a gradual loss of precision when the result of an operation is not exactly zero but is too close to zero to be represented by a normalized number.

A denormal number is represented with a biased exponent of all 0 bits, which represents an exponent of −126 in single precision (not −127), or −1022 in double precision (not −1023). [3] In contrast, the smallest biased exponent representing a normal number is 1 (see examples below).

Representation of non-numbers

The biased-exponent field is filled with all 1 bits to indicate either infinity or an invalid result of a computation.

Positive and negative infinity

Positive and negative infinity are represented thus:

sign = 0 for positive infinity, 1 for negative infinity.
biased exponent = all 1 bits.
fraction = all 0 bits.

NaN

Some operations of floating-point arithmetic are invalid, such as taking the square root of a negative number. The act of reaching an invalid result is called a floating-point exception. An exceptional result is represented by a special code called a NaN, for "Not a Number". All NaNs in IEEE 754-1985 have this format:

sign = either 0 or 1.
biased exponent = all 1 bits.
fraction = anything except all 0 bits (since all 0 bits represents infinity).

Range and precision

Relative precision of single (binary32) and double precision (binary64) numbers, compared with decimal representations using a fixed number of significant digits. Relative precision is defined here as ulp(x)/x, where ulp(x) is the unit in the last place in the representation of x, i.e. the gap between x and the next representable number. IEEE 754 relative precision.svg
Relative precision of single (binary32) and double precision (binary64) numbers, compared with decimal representations using a fixed number of significant digits. Relative precision is defined here as ulp(x)/x, where ulp(x) is the unit in the last place in the representation of x, i.e. the gap between x and the next representable number.

Precision is defined as the minimum difference between two successive mantissa representations; thus it is a function only in the mantissa; while the gap is defined as the difference between two successive numbers. [4]

Single precision

Single-precision numbers occupy 32 bits. In single precision:

Some example range and gap values for given exponents in single precision:

Actual Exponent (unbiased)Exp (biased)MinimumMaximumGap
−11260.5≈ 0.999999940395≈ 5.96046e-8
01271≈ 1.999999880791≈ 1.19209e-7
11282≈ 3.999999761581≈ 2.38419e-7
21294≈ 7.999999523163≈ 4.76837e-7
101371024≈ 2047.999877930≈ 1.22070e-4
111382048≈ 4095.999755859≈ 2.44141e-4
231508388608167772151
2415116777216335544302
127254≈ 1.70141e38≈ 3.40282e38≈ 2.02824e31

As an example, 16,777,217 cannot be encoded as a 32-bit float as it will be rounded to 16,777,216. However, all integers within the representable range that are a power of 2 can be stored in a 32-bit float without rounding.

Double precision

Double-precision numbers occupy 64 bits. In double precision:

Some example range and gap values for given exponents in double precision:

Actual Exponent (unbiased)Exp (biased)MinimumMaximumGap
−110220.5≈ 0.999999999999999888978≈ 1.11022e-16
010231≈ 1.999999999999999777955≈ 2.22045e-16
110242≈ 3.999999999999999555911≈ 4.44089e-16
210254≈ 7.999999999999999111822≈ 8.88178e-16
1010331024≈ 2047.999999999999772626≈ 2.27374e-13
1110342048≈ 4095.999999999999545253≈ 4.54747e-13
521075450359962737049690071992547409911
5310769007199254740992180143985094819822
10232046≈ 8.98847e307≈ 1.79769e308≈ 1.99584e292

Extended formats

The standard also recommends extended format(s) to be used to perform internal computations at a higher precision than that required for the final result, to minimise round-off errors: the standard only specifies minimum precision and exponent requirements for such formats. The x87 80-bit extended format is the most commonly implemented extended format that meets these requirements.

Examples

Here are some examples of single-precision IEEE 754 representations:

TypeSignActual ExponentExp (biased)Exponent fieldFraction fieldValue
Zero0−12600000 0000000 0000 0000 0000 0000 00000.0
Negative zero 1−12600000 0000000 0000 0000 0000 0000 00000.0
One001270111 1111000 0000 0000 0000 0000 00001.0
Minus One101270111 1111000 0000 0000 0000 0000 00001.0
Smallest denormalized number *−12600000 0000000 0000 0000 0000 0000 0001±223× 2126 = ±2149 ≈ ±1.4×10−45
"Middle" denormalized number*−12600000 0000100 0000 0000 0000 0000 0000±21× 2126 = ±2127 ≈ ±5.88×10−39
Largest denormalized number*−12600000 0000111 1111 1111 1111 1111 1111±(1−2−23) × 2−126 ≈ ±1.18×10−38
Smallest normalized number*−12610000 0001000 0000 0000 0000 0000 0000±2−126 ≈ ±1.18×10−38
Largest normalized number*1272541111 1110111 1111 1111 1111 1111 1111±(22−23) × 2127 ≈ ±3.4×1038
Positive infinity01282551111 1111000 0000 0000 0000 0000 0000+∞
Negative infinity11282551111 1111000 0000 0000 0000 0000 0000−∞
Not a number *1282551111 1111non zeroNaN
* Sign bit can be either 0 or 1 .

Comparing floating-point numbers

Every possible bit combination is either a NaN or a number with a unique value in the affinely extended real number system with its associated order, except for the two combinations of bits for negative zero and positive zero, which sometimes require special attention (see below). The binary representation has the special property that, excluding NaNs, any two numbers can be compared as sign and magnitude integers (endianness issues apply). When comparing as 2's-complement integers: If the sign bits differ, the negative number precedes the positive number, so 2's complement gives the correct result (except that negative zero and positive zero should be considered equal). If both values are positive, the 2's complement comparison again gives the correct result. Otherwise (two negative numbers), the correct FP ordering is the opposite of the 2's complement ordering.

Rounding errors inherent to floating point calculations may limit the use of comparisons for checking the exact equality of results. Choosing an acceptable range is a complex topic. A common technique is to use a comparison epsilon value to perform approximate comparisons. [6] Depending on how lenient the comparisons are, common values include 1e-6 or 1e-5 for single-precision, and 1e-14 for double-precision. [7] [8] Another common technique is ULP, which checks what the difference is in the last place digits, effectively checking how many steps away the two values are. [9]

Although negative zero and positive zero are generally considered equal for comparison purposes, some programming language relational operators and similar constructs treat them as distinct. According to the Java Language Specification, [10] comparison and equality operators treat them as equal, but Math.min() and Math.max() distinguish them (officially starting with Java version 1.1 but actually with 1.1.1), as do the comparison methods equals(), compareTo() and even compare() of classes Float and Double.

Rounding floating-point numbers

The IEEE standard has four different rounding modes; the first is the default; the others are called directed roundings .

Extending the real numbers

The IEEE standard employs (and extends) the affinely extended real number system, with separate positive and negative infinities. During drafting, there was a proposal for the standard to incorporate the projectively extended real number system, with a single unsigned infinity, by providing programmers with a mode selection option. In the interest of reducing the complexity of the final standard, the projective mode was dropped, however. The Intel 8087 and Intel 80287 floating point co-processors both support this projective mode. [11] [12] [13]

Functions and predicates

Standard operations

The following functions must be provided:

History

In 1976, Intel was starting the development of a floating-point coprocessor. [15] [16] Intel hoped to be able to sell a chip containing good implementations of all the operations found in the widely varying maths software libraries. [15] [17]

John Palmer, who managed the project, believed the effort should be backed by a standard unifying floating point operations across disparate processors. He contacted William Kahan of the University of California, who had helped improve the accuracy of Hewlett-Packard's calculators. Kahan suggested that Intel use the floating point of Digital Equipment Corporation's (DEC) VAX. The first VAX, the VAX-11/780 had just come out in late 1977, and its floating point was highly regarded. However, seeking to market their chip to the broadest possible market, Intel wanted the best floating point possible, and Kahan went on to draw up specifications. [15] Kahan initially recommended that the floating point base be decimal [18] [ unreliable source? ] but the hardware design of the coprocessor was too far along to make that change.

The work within Intel worried other vendors, who set up a standardization effort to ensure a "level playing field". Kahan attended the second IEEE 754 standards working group meeting, held in November 1977. He subsequently received permission from Intel to put forward a draft proposal based on his work for their coprocessor; he was allowed to explain details of the format and its rationale, but not anything related to Intel's implementation architecture. The draft was co-written with Jerome Coonen and Harold Stone, and was initially known as the "Kahan-Coonen-Stone proposal" or "K-C-S format". [15] [16] [17] [19]

As an 8-bit exponent was not wide enough for some operations desired for double-precision numbers, e.g. to store the product of two 32-bit numbers, [20] both Kahan's proposal and a counter-proposal by DEC therefore used 11 bits, like the time-tested 60-bit floating-point format of the CDC 6600 from 1965. [16] [19] [21] Kahan's proposal also provided for infinities, which are useful when dealing with division-by-zero conditions; not-a-number values, which are useful when dealing with invalid operations; denormal numbers, which help mitigate problems caused by underflow; [19] [22] [23] and a better balanced exponent bias, which can help avoid overflow and underflow when taking the reciprocal of a number. [24] [25]

Even before it was approved, the draft standard had been implemented by a number of manufacturers. [26] [27] The Intel 8087, which was announced in 1980, was the first chip to implement the draft standard.

Intel 8087 floating-point coprocessor Intel C8087.jpg
Intel 8087 floating-point coprocessor

In 1980, the Intel 8087 chip was already released, [28] but DEC remained opposed, to denormal numbers in particular, because of performance concerns and since it would give DEC a competitive advantage to standardise on DEC's format.

The arguments over gradual underflow lasted until 1981 when an expert hired by DEC to assess it sided against the dissenters. DEC had the study done in order to demonstrate that gradual underflow was a bad idea, but the study concluded the opposite, and DEC gave in. In 1985, the standard was ratified, but it had already become the de facto standard a year earlier, implemented by many manufacturers. [16] [19] [5]

See also

Notes

  1. Precision: The number of decimal digits precision is calculated via number_of_mantissa_bits * Log10(2). Thus ~7.2 and ~15.9 for single and double precision respectively.

Related Research Articles

<span class="mw-page-title-main">Floating-point arithmetic</span> Computer approximation for real numbers

In computing, floating-point arithmetic (FP) is arithmetic that represents subsets of real numbers using an integer with a fixed precision, called the significand, scaled by an integer exponent of a fixed base. Numbers of this form are called floating-point numbers. For example, 12.345 is a floating-point number in base ten with five digits of precision:

In computing, NaN, standing for Not a Number, is a particular value of a numeric data type which is undefined as a number, such as the result of 0/0. Systematic use of NaNs was introduced by the IEEE 754 floating-point standard in 1985, along with the representation of other non-finite quantities such as infinities.

Double-precision floating-point format is a floating-point number format, usually occupying 64 bits in computer memory; it represents a wide range of numeric values by using a floating radix point.

In computer science, subnormal numbers are the subset of denormalized numbers that fill the underflow gap around zero in floating-point arithmetic. Any non-zero number with magnitude smaller than the smallest positive normal number is subnormal, while denormal can also refer to numbers outside that range.

<span class="mw-page-title-main">Rounding</span> Replacing a number with a simpler value

Rounding or rounding off means replacing a number with an approximate value that has a shorter, simpler, or more explicit representation. For example, replacing $23.4476 with $23.45, the fraction 312/937 with 1/3, or the expression √2 with 1.414.

The IEEE Standard for Floating-Point Arithmetic is a technical standard for floating-point arithmetic originally established in 1985 by the Institute of Electrical and Electronics Engineers (IEEE). The standard addressed many problems found in the diverse floating-point implementations that made them difficult to use reliably and portably. Many hardware floating-point units use the IEEE 754 standard.

The significand is the first (left) part of a number in scientific notation or related concepts in floating-point representation, consisting of its significant digits. Depending on the interpretation of the exponent, the significand may represent an integer or a fractional number.

Hexadecimal floating point is a format for encoding floating-point numbers first introduced on the IBM System/360 computers, and supported on subsequent machines based on that architecture, as well as machines which were intended to be application-compatible with System/360.

In computing, signed number representations are required to encode negative numbers in binary number systems.

Signed zero is zero with an associated sign. In ordinary arithmetic, the number 0 does not have a sign, so that −0, +0 and 0 are equivalent. However, in computing, some number representations allow for the existence of two zeros, often denoted by −0 and +0, regarded as equal by the numerical comparison operations but with possible different behaviors in particular operations. This occurs in the sign-magnitude and ones' complement signed number representations for integers, and in most floating-point number representations. The number 0 is usually encoded as +0, but can still be represented by +0, −0, or 0.

In computing, minifloats are floating-point values represented with very few bits. This reduced precision makes them ill-suited for general-purpose numerical calculations, but they are useful for special purposes such as:

Extended precision refers to floating-point number formats that provide greater precision than the basic floating-point formats. Extended precision formats support a basic format by minimizing roundoff and overflow errors in intermediate values of expressions on the base format. In contrast to extended precision, arbitrary-precision arithmetic refers to implementations of much larger numeric types using special software.

Decimal floating-point (DFP) arithmetic refers to both a representation and operations on decimal floating-point numbers. Working directly with decimal (base-10) fractions can avoid the rounding errors that otherwise typically occur when converting between decimal fractions and binary (base-2) fractions.

IEEE 754-2008 is a revision of the IEEE 754 standard for floating-point arithmetic. It was published in August 2008 and is a significant revision to, and replaces, the IEEE 754-1985 standard. The 2008 revision extended the previous standard where it was necessary, added decimal arithmetic and formats, tightened up certain areas of the original standard which were left undefined, and merged in IEEE 854 . In a few cases, where stricter definitions of binary floating-point arithmetic might be performance-incompatible with some existing implementation, they were made optional. In 2019, it was updated with a minor revision IEEE 754-2019.

In computing, half precision is a binary floating-point computer number format that occupies 16 bits in computer memory. It is intended for storage of floating-point values in applications where higher precision is not essential, in particular image processing and neural networks.

In computing, quadruple precision is a binary floating-point–based computer number format that occupies 16 bytes with precision at least twice the 53-bit double precision.

Single-precision floating-point format is a computer number format, usually occupying 32 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point.

In computing, Microsoft Binary Format (MBF) is a format for floating-point numbers which was used in Microsoft's BASIC languages, including MBASIC, GW-BASIC and QuickBASIC prior to version 4.00.

In computing, octuple precision is a binary floating-point-based computer number format that occupies 32 bytes in computer memory. This 256-bit octuple precision is for applications requiring results in higher than quadruple precision.

The bfloat16 floating-point format is a computer number format occupying 16 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point. This format is a shortened (16-bit) version of the 32-bit IEEE 754 single-precision floating-point format (binary32) with the intent of accelerating machine learning and near-sensor computing. It preserves the approximate dynamic range of 32-bit floating-point numbers by retaining 8 exponent bits, but supports only an 8-bit precision rather than the 24-bit significand of the binary32 format. More so than single-precision 32-bit floating-point numbers, bfloat16 numbers are unsuitable for integer calculations, but this is not their intended use. Bfloat16 is used to reduce the storage requirements and increase the calculation speed of machine learning algorithms.

References

  1. IEEE Standard for Binary Floating-Point Arithmetic. 1985. doi:10.1109/IEEESTD.1985.82928. ISBN   0-7381-1165-1.
  2. "ANSI/IEEE Std 754-2019". 754r.ucbtest.org. Retrieved 2019-08-06.
  3. Hennessy (2009). Computer Organization and Design . Morgan Kaufmann. p.  270. ISBN   9780123744937.
  4. Hossam A. H. Fahmy; Shlomo Waser; Michael J. Flynn, Computer Arithmetic (PDF), archived from the original (PDF) on 2010-10-08, retrieved 2011-01-02
  5. 1 2 3 William Kahan (October 1, 1997). "Lecture Notes on the Status of IEEE 754" (PDF). University of California, Berkeley. Retrieved 2007-04-12.
  6. "Godot math_funcs.h". GitHub.com. 30 July 2022.
  7. "Godot math_defs.h". GitHub.com. 30 July 2022.
  8. "Godot MathfEx.cs". GitHub.com.
  9. "Comparing Floating Point Numbers, 2012 Edition". randomascii.wordpress.com. 26 February 2012.
  10. "Java Language and Virtual Machine Specifications". Java Documentation.
  11. John R. Hauser (March 1996). "Handling Floating-Point Exceptions in Numeric Programs" (PDF). ACM Transactions on Programming Languages and Systems. 18 (2): 139–174. doi: 10.1145/227699.227701 . S2CID   9820157.
  12. David Stevenson (March 1981). "IEEE Task P754: A proposed standard for binary floating-point arithmetic". IEEE Computer. 14 (3): 51–62. doi:10.1109/C-M.1981.220377. S2CID   15523399.
  13. William Kahan and John Palmer (1979). "On a proposed floating-point standard". SIGNUM Newsletter. 14 (Special): 13–21. doi:10.1145/1057520.1057522. S2CID   16981715.
  14. ISO/IEC 9899:1999 - Programming languages - C. Iso.org. §7.12.14.
  15. 1 2 3 4 "Intel and Floating-Point - Updating One of the Industry's Most Successful Standards - The Technology Vision for the Floating-Point Standard" (PDF). Intel. 2016. Archived from the original (PDF) on 2016-03-04. Retrieved 2016-05-30. (11 pages)
  16. 1 2 3 4 "An Interview with the Old Man of Floating-Point". cs.berkeley.edu. 1998-02-20. Retrieved 2016-05-30.
  17. 1 2 Woehr, Jack, ed. (1997-11-01). "A Conversation with William Kahan". Dr. Dobb's . drdobbs.com. Retrieved 2016-05-30.
  18. W. Kahan 2003, pers. comm. to Mike Cowlishaw and others after an IEEE 754 meeting
  19. 1 2 3 4 "IEEE 754: An Interview with William Kahan" (PDF). dr-chuck.com. Retrieved 2016-06-02.
  20. "IEEE vs. Microsoft Binary Format; Rounding Issues (Complete)". Microsoft Support. Microsoft. 2006-11-21. Article ID KB35826, Q35826. Archived from the original on 2020-08-28. Retrieved 2010-02-24.
  21. Thornton, James E. (1970). Written at Advanced Design Laboratory, Control Data Corporation. Design of a Computer: The Control Data 6600 (PDF) (1 ed.). Glenview, Illinois, USA: Scott, Foresman and Company. LCCN   74-96462. Archived (PDF) from the original on 2020-08-28. Retrieved 2016-06-02. (1+13+181+2+2 pages)
  22. Kahan, William Morton. "Why do we need a floating-point arithmetic standard?" (PDF). cs.berkeley.edu. Retrieved 2016-06-02.
  23. Kahan, William Morton; Darcy, Joseph D. "How Java's Floating-Point Hurts Everyone Everywhere" (PDF). cs.berkeley.edu. Retrieved 2016-06-02.
  24. Turner, Peter R. (2013-12-21). Numerical Analysis and Parallel Processing: Lectures given at The Lancaster …. Springer. ISBN   978-3-66239812-8 . Retrieved 2016-05-30.
  25. "Names for Standardized Floating-Point Formats" (PDF). cs.berkeley.edu. Retrieved 2016-06-02.
  26. Charles Severance (20 February 1998). "An Interview with the Old Man of Floating-Point".
  27. Charles Severance. "History of IEEE Floating-Point Format". Connexions. Archived from the original on 2009-11-20.
  28. "Molecular Expressions: Science, Optics & You - Olympus MIC-D: Integrated Circuit Gallery - Intel 8087 Math Coprocessor". micro.magnet.fsu.edu. Retrieved 2016-05-30.

Further reading