The Nikon Expeed image/video processors (often styled EXPEED) are media processors for Nikon's digital cameras. They perform a large number of tasks: Bayer filtering, demosaicing, image sensor corrections/dark-frame subtraction, image noise reduction, image sharpening, image scaling, gamma correction, image enhancement/Active D-Lighting, colorspace conversion, chroma subsampling, framerate conversion, lens distortion/chromatic aberration correction, image compression/JPEG encoding, video compression, display/video interface driving, digital image editing, face detection, audio processing/compression/encoding and computer data storage/data transmission.
Expeed's multi-processor system on a chip solution integrates an image processor in multi-core processor architecture, with each single processor-core able to compute many instructions/operations in parallel. Storage and display interfaces and other modules are added and a digital signal processor (DSP) increases the number of simultaneous computations. An on-chip 32-bit microcontroller initiates and controls the operation and data transfers of all processors, modules and interfaces and can be seen as the main control unit of the camera.
In each generation Nikon uses different versions for its professional and consumer DSLRs / MILCs, whereas its compact cameras use totally different architectures. This is different from for example Canons DIGIC: its professional DSLRs double the processors of its consumer DSLR series. The Expeed is an application-specific integrated circuit (ASIC) built by Socionext specifically for Nikon designs according to Nikon specifications.
The Nikon Expeed is based on the Socionext Milbeaut imaging processors with 16-bit per pixel [1] multi-core FR-V processor architecture, using a highly parallel pipelined architecture which allows efficient hardware use, increasing throughput and reducing power consumption. Each core uses an eight-way 256-bit very long instruction word (VLIW, MIMD) and is organized in a four-unit superscalar pipelined architecture (Integer (ALU)-, Floating-point- and two media-processor-units) giving a peak performance of up to 28 instructions per clock cycle and core. Due to the used four-way single instruction, multiple data (SIMD) vector processor units, data is processed with up to 112 data operations per cycle and core. [2]
An on-chip 32-bit Fujitsu FR RISC microcontroller core is used to initiate and control all processors, modules and interfaces. [3] [4] [5] [6] [7] The Expeed versions designated EI-14x and the Expeed 2 and 3 additionally include a HD video codec engine (FR-V based) and a 16-bit DSP with separate on-chip 4-block Harvard RAM which is usable for example for additional image- and audio-processing. The Expeed 3 (FR) (EI-158/175) is based on an improved Expeed 2 EI-154 with greatly increased processing speed.
A new architecture in the Expeed 3 (ARM) offers a highly increased speed in its image processor (with even two pipelines on the EI-160), its H.264 video encoder and is controlled by a dual-core ARM architecture microcontroller replacing the Fujitsu FR. [8] [9]
CMOS/CCD image sensors are connected with additional external ICs, Nikon drivers, with the exception of the Nikon D3100. [5] This is done by a mixed analog/digital interface which controls the sensor digitally, but receives analog signals with parallel 14-bit analog-to-digital (A/D) converters. The Expeed variants EI-142 and EI-158 use Nikon ASICs to connect all full-frame (FX) digital SLR sensors and additionally the Nikon D300/D300s with 12 simultaneous, parallel analog signal readout channels. Mainly due to a larger settling time it allows improved conversion accuracy compared to the four channels in the previous Nikon D2X / D2Xs, Nikon D200 or the Canon EOS 5D Mark II. [11] [12] [13] six-channel readout is supported by the EI-149 and EI-154 used on the D90, D5000, D7000 and D5100.
The D3100 uses an Analog Devices sensor interface with integrated analog-to-digital converters. The result is a dynamic range only at the level of competitors like the (higher priced) Canon EOS 600D; [14] lower than other Nikon DSLRs with the same Expeed 2 variant.
The Expeed EI-15x and EI-17x A/D converters allow an increased image sensor readout clock frequency with improved A/D converter accuracy, especially when using 14-bit sampling. Expeed A/D converters used for EI-149 or all EI-142 need considerably reduced clock rates (1.8 fps on Nikon D3X) for higher accuracy, limiting for example the D3s dynamic range at low ISOs. [15] [16] [17]
The Expeed 3 (ARM), first used in the Nikon 1 series, connects a data stream with 24 digital channels (bus), using A/D converters integrated on the image sensor chip. [18]
The first variant, EI-14x originally included a video encoder capable of processing VGA resolution with 30 frames per second and MPEG-4 encoding. [4] The software based video processor realized with FR-V processors enabled a reprogramming: [2] [19] [20] By using Motion JPEG encoding with 24p frame rate, Nikon achieved 720p HD video resolution. The advantages are easy JPEG image extraction, no motion compensation artifacts and low processing power enabling higher resolution, and the disadvantage is a larger file size, nearly reaching the 2 GB limit (for full compatibility) in 5 minutes. The Nikon D90 was the first DSLR with video recording capabilities.
The Expeed 2 (variant EI-154) greatly expanded the capabilities by its 1080p H.264/MPEG-4 AVC HD video encoder. It also offers an increased image sensor analog signal readout clock rate, reducing rolling shutter. [15] [16] [17] [21]
Compared to competitors from Canon (DIGIC, "IPP" compression – MPEG-2 [22] equivalent: Only one previous frame analyzed [23] [24] ) the Expeed 2 offers video compression also based on complex B-frames (bi-directional differencing between frames and motion prediction), which has the advantage of higher quality (lower motion blur based on better motion compensation) even with significant higher compression ratio. [25] This compression requires considerably higher computing power. [26] In 2012 the Canon 5D Mark III introduced a similar compression called "IPB". [23] Also introduced was "All-I", which uses the simpler I‑frames (coded pictures) without processing any differences between them, but using a low compression (high data rate) and behaves also in other uses like image extraction quite similar to Motion JPEG formerly used by Nikon. [27]
The Expeed 3 (FR) (variants EI-158 and EI-175) offers no significant change, but introduced the first DSLRs to offer uncompressed video output (8bit 4:2:2) over HDMI: Nikon D4, Nikon D800/D800E, Nikon D600, Nikon D7100 and Nikon D5200. The Expeed 3 (ARM) introduced high-speed video (slow-motion) in its enhanced H.264 HD video engine.
Socionext specifies each Milbeaut generation with different numbers of processors. Nikon gives no details, but uses different designated processors in its professional and consumer lines. Although the Milbeaut (Expeed) is used in different Nikon designs and by other manufacturers, the software/firmware specifies many of its functions and details and the number of processors or included modules may vary in this ASIC.
DSLRs announced before August 2006 do not contain processors named Expeed (for example Nikon D70/D70s processor: EI-118), although that does not mean that these processors use a different architecture. [28] Or the Nikon D200 processor (EI-126) uses the same, but greatly expanded firmware as the D80 (unofficial "Expeed").
The Expeed processor variant EI-137 is found in the Nikon D40, Nikon D40x and Nikon D80 –as it is officially in the later-released Nikon D60 and Nikon D3000.
First used in the Nikon D3 and Nikon D300 in 2007, the Expeed was used later in the Nikon D3X, Nikon D700 and Nikon D300s, marked EI-142, and the consumer line variant with reduced processor cores in the Nikon D90 and Nikon D5000, marked EI-149. It is based on a Socionext Milbeaut imaging-processor with 720p Motion JPEG video encoder, DSP and FR-80 (EI-14x versions) core. It uses a 90 nanometer process technology.
The EI-137 variant in the Nikon D60 and Nikon D3000 –additionally found in the Nikon D40, Nikon D40x and Nikon D80 [6] –is based on the older Milbeaut M-3 in 180 nanometer technology (like all former Expeed/Milbeaut since 2001). It includes a FR-71 core with only 12-bit, two-channel image sensor readout, no DSP, slower memory and has a reduced feature set. [29]
1080p H.264/MPEG-4 HD video encoder, improved face detection, image noise reduction and image distortion correction [30] are the main improved features compared to EI-14x Expeeds. [3] Used in the Nikon D7000, [31] Nikon D3100 and Nikon D5100 and Nikon marked EI-154. [32] Although image sensor readout clock frequency has increased by a factor of 1.75, A/D converter accuracy is improved, especially when using 14-bit. Image processor performance is increased, performing a higher continuous shooting frame rate even when high ISO noise reduction or Active D-Lighting is activated. The Expeed EI-15x are controlled by an integrated FR-80/FR-81 core. Power consumption is also decreased by the Socionext 65 nm process.
The Nikon D3s processor [33] –although named Expeed 2 –uses nearly the same EI-142 processor as the previous D3/D3X [34] featuring for example only 720p Motion JPEG video. It offers the same image sensor interface with identical speed and A/D converter accuracy, limiting the D3s dynamic range at 200 and especially 100 ISOs lower as the D7000/D5100. [15] [16] [35]
Variant used in some Coolpix compact cameras. Cheaper Nikon compact cameras use Sanyo or Zoran Coach image/video processors; both with a completely different technology and different firmware compared to the Expeed. [6]
Compared to the previous Expeed 1 (EI-142), it offers the same improvements as the Expeed 2 EI-154 with Socionext 65 nm process, including increased A/D converter accuracy and image sensor analog signal readout clock rate, reducing rolling shutter. [15] [16] [17] [21] Computing power is highly increased. Marked EI-158, this variant is used by the Nikon D4 and Nikon D800. [36] [37] The EI-158 was the first Expeed to offer uncompressed video output (8bit 4:2:2) over HDMI.
The Nikon D600 (teardown [38] ), Nikon D3200 and Nikon D5200 (teardown [39] ) use an Expeed 3 (EI-175, differently marked ML-1131 on D5200), which is, according to Nikon, the same as used for the D4 and D800 series. [5] [40] Its architecture is close to the Expeed 2 variant EI-154 with some improvements like DDR3 memory, and with increased computing power. The D5200 uses a package on package with a 4 Gbit DDR3 SDRAM on top.
In the Nikon 1 series September 2011 Nikon introduced a new largely changed architecture – the main control unit it uses an ARM microcontroller which requires new firmware compared to the totally different Fujitsu FR microcontroller used in all former Milbeaut and Expeed processors. It is also the first Expeed using digital image sensor readout – no analog image sensor interface is needed. In the Nikon 1 cameras introduced September 2011 it uses 1 GB fast DDR2 RAM packaged in 2x 4 Gbit chips. Nikon marked EI-160, manufactured in the Socionext 65 nm process.
High-speed dual multi-core image-processing engines with world record (Nikon claim) 600 megapixels per second speed, enhanced H.264 HD video engine and controlled by a dual-core ARM microcontroller are the main improvements. [9] [18] [41] Its high speed allows the world's fastest speed (Nikon claim) of 60 frames per second (10 fps with full autofocus).
The Expeed 3A, a successor to the Expeed 3 EI-160 used in the Nikon 1 series, was first released in the Nikon 1 V2 and mainly features an increased world record image-processing speed of up to 850 megapixels per second. [42] This enables 60 frames per second (15 fps with full autofocus) speed even with the new 14 megapixel image sensor. It is developed exclusively for Nikon 1 cameras. [43]
Expeed 4 uses a processor with ARM central controller, and is used in the Nikon D810, Nikon D750, Nikon D5300, Nikon D5500, Nikon D5600, Nikon D3300, Nikon D3400, Nikon D3500 and Nikon D7200. It offers full HD (1080p) video capture at 50/60 fps with improved contrast detection autofocus and live preview autofocus. It includes all of the features of the Expeed 3 (FR) and older versions of Expeed, plus it consumes less power.
The Nikon D4S's processor is identical to the Nikon D4, marked EI-158, using its processing power with improved software enabling 1080p video capture at 50/60fps, 11 fps photos with improved autofocus, new noise reduction with image-content variable algorithm (context-adaptive) [44] [45] and other improvements. [46] [47]
Version 4A is used in the Nikon 1 V3, Nikon 1 J4 and Nikon 1 S2.
Nikon announced EXPEED 5 processor engine in their new DX and FX cameras Nikon D500 and Nikon D5 at CES 2016, (Las Vegas, January 5, 2016) [48] and also used in the Nikon D7500 and D850. [49]
The EXPEED 5A image-processing engine is first seen in the Nikon 1 J5, which was announced April 2, 2015. [50] In the Nikon 1 J5 it is capable of 4k Ultra HD (3840*2160) at 15 fps, Full-HD (1920*1080) at 60 fps, HD (1280*720) at 120 fps, 800 x 296 at 400 fps and 400 x 144 at 1200 fps. It can handle 20MP burst photos at 20 fps with autofocus at each frame, and even 60 fps with autofocus fixed at the first frame, but note that the buffer size is yet unknown and probably very small.
Nikon announced the Expeed 6 processor engine on 23 August 2018. [51] It appears in their Nikon Z 7, Nikon Z 6, Nikon Z 5, Nikon Z 50, Nikon Z 30, Nikon Z fc, Nikon D780 and Nikon D6 cameras. The Nikon Z 6II and Nikon Z 7II each have two Expeed 6 processors. [51] [52]
Nikon announced the Expeed 7 processor engine on 28 October 2021. It appears in the Nikon Z 50II, Nikon Z 6III, Nikon Z 8, Nikon Z 9, and Nikon Z f mirrorless cameras. Expeed 7 has 10 times the image processing speed of its predecessor and is powerful enough for the camera to omit a dedicated autofocus engine. [53] It can handle complex AF and AE calculations at 120 cycles per second, separately processes dual-streamed data from stacked image sensors, and enables features like 120 fps still shooting, internal 8k RAW video recording and a blackout-free viewing experience. [54]
As input/output (I/O) processor Nikon uses external 32-bit microcontrollers to connect additional sensors and displays: Viewfinder, top-display, flash (Speedlight)/hot shoe, shutter/aperture motors, metering/autofocus sensors and lens/battery grip/battery control. Used variants are the Fujitsu FR in the Expeed EI-14x series, which changed to MIPS architecture in the Expeed EI-15x and EI-17x series. The professional series uses two or more Hitachi/Renesas H8SX controllers. Former DSLRs used H8S microcontrollers. [55]
The ARM-based Expeed in the Nikon 1 series with its dual ARM core needs no additional I/O processors. [56] The Nikon 1 series also includes an Epson graphic processor.
As with CHDK or Magic Lantern modified Canon digital cameras based on DIGIC processors, a group of programmers called "Nikon Hacker" develops custom firmware, [5] making recent progress including an FR emulator for some DSLRs. [57] [58] [59] It was shown that Nikon uses the Softune integrated development environment together with an μITRON realtime kernel. [28] Currently there is some modified firmware available mainly removing time based video and uncompressed NEF files restrictions, [60] but there is no stable alternative firmware available, as the project is still in an early state.
The Nikon supplied firmware-updates normally include the firmware A for the I/O processor and the firmware B to control Expeeds by integrated FR microcontrollers (different for the ARM-based Expeed 3).
Besides a general analysis of the hardware and software of the D7000, D5100 and D3100 and newer cameras, [61] the project focuses on:
The changelog [63] shows firmware also for newer variants like D800, D610, D5200 and D3200.
This was not a firmware hack, but a method (Photopc – digital camera control [64] ) calling an already implemented diagnostic mode on some old Nikon Coolpix cameras with Fujitsu Sparclite processors. Using this mode, they could write raw images.
The Fujitsu FR-V is one of the very few processors ever able to process both a very long instruction word (VLIW) and vector processor instructions at the same time, increasing throughput with high parallel computing while increasing performance per watt and hardware efficiency. The family was presented in 1999. Its design was influenced by the VPP500/5000 models of the Fujitsu VP/2000 vector processor supercomputer line.
The Nikon D3100 is a 14.2-megapixel DX format DSLR Nikon F-mount camera announced by Nikon on August 19, 2010. It replaced the D3000 as Nikon's entry level DSLR. It introduced Nikon's new EXPEED 2 image processor and was the first Nikon DSLR featuring full high-definition video recording with full-time autofocus and H.264 compression, instead of Motion JPEG compression. It was also the first Nikon DSLR to provide high-definition video recording at more than one frame rate.
The Nikon D5100 is a 16.2-megapixel DX-format DSLR F-mount camera announced by Nikon on April 5, 2011. It features the same 16.2-megapixel CMOS sensor as the D7000 with 14-bit depth, while delivering Full HD 1080p video mode at either 24, 25 or 30 fps. The D5100 is the first Nikon DSLR to offer 1080p video at a choice of frame rates; previous Nikon DSLRs that recorded 1080p only did so at 24 fps. It replaced the D5000 and was replaced by the D5200.
The Nikon 1 series is a discontinued camera line from Nikon, originally announced on 21 September 2011. The cameras utilized Nikon 1-mount lenses, and featured 1" CX format sensors.
The Nikon D800 is a 36.3-megapixel professional-grade full-frame digital single-lens reflex camera produced by Nikon Corporation. It was given a Gold Award by Digital Photography Review.
The Nikon D4 is a 16.2-megapixel professional-grade full frame (35mm) digital single-lens reflex camera (DSLR) announced by Nikon Corporation on 6 January 2012. It succeeds the Nikon D3S and introduces a number of improvements including a 16.2 megapixel sensor, improved auto-focus and metering sensors and the ability to shoot at an extended ISO speed of 204,800. The camera was released in February 2012 at a recommended retail price of $5999.95. It is the first camera to use the new XQD memory cards. It was replaced by the Nikon D4S as Nikon's flagship camera.
The Fujitsu FR is a 32-bit RISC processor family. New variants include a floating point unit and partly video input analog-to-digital converter and digital signal processor. It is supported by Softune, GNU Compiler Collection and other integrated development environments.
The Nikon D3200 is a 24.2-megapixel DX format DSLR Nikon F-mount camera officially launched by Nikon on April 19, 2012. It is marketed as an entry-level DSLR camera for beginners and experienced DSLR hobbyists who are ready for more advanced specs and performance.
The Nikon D600 is a 24.3-effective-megapixel FX-format full-frame digital SLR camera from Nikon released on September 13, 2012 targeted at professionals and enthusiasts. It began shipping on September 18, 2012; at introduction, its suggested retail price in the U.S. was $2099 for the body only and $2699 with a 24–85 mm kit lens. The Nikon D600 was given a Gold Award by Digital Photography Review.
The Nikon 1 V2 is a Nikon 1 series high-speed mirrorless interchangeable-lens camera launched by Nikon on October 24, 2012.
The Nikon D5200 is an F-mount DSLR camera with a newly developed 24.1-megapixel DX-format CMOS image sensor first announced by Nikon on November 6, 2012 for most of the world and January 7, 2013 for the North American market.
The Socionext Milbeaut image/video processors are media processors in multi-processor system on a chip architecture. Started by Fujitsu with the M-1 Series in 2000 each generation has several variants regarding included modules and processor-cores, built for mobile phones, digital compact cameras, MILCs and DSLRs like Leica M and Leica S2, Nikon DSLRs, some Pentax K mount cameras and for the Sigma True-II processor.
The Nikon D7100 is a 24.1-megapixel digital single-lens reflex camera model announced by Nikon in February 2013. It is a 'prosumer' model that replaces the Nikon D7000 as Nikon's flagship DX-format camera, fitting between the company's entry-level and professional DSLR models. This camera is the first ever from Nikon with no optical low-pass filter incorporated. At launch, Nikon gave the D7100 estimated selling price in the United States as US$ 949.95 for the body.
Nikon D3300 is a 24.2-megapixel DX format DSLR Nikon F-mount camera officially launched by Nikon on 7 January 2014. It was marketed as an entry-level DSLR camera for beginners and experienced DSLR hobbyist who were ready for more advanced specs and performance. It replaced the D3200 as Nikon's entry level DSLR. The D3300 usually came with an 18-55mm VR II kit lens, which is the upgraded model of older VR lens. The new kit lens has the ability to retract its barrel, shortening it for easy storage.
The Nikon D810 is a 36.3-megapixel professional-grade full-frame digital single-lens reflex camera produced by Nikon. The camera was officially announced in June 2014, and became available in July 2014.
The Nikon D750 is a full-frame DSLR camera announced by Nikon on September 12, 2014. It is an extensive upgrade from the D610, but with the same general body and control characteristics, along with 24 megapixel resolution. Despite the 7, there is little relationship with the D700, which was the precursor to the D800. The D600 and D610 evolved as a full-frame consumer cameras with similar structure and controls to the D7000 series of cropped frame cameras. The D750 shares similar structure and controls with the cropped-frame D7500.
The Nikon D5 is a full frame professional DSLR camera announced by Nikon Corporation on 6 January 2016 to succeed the D4S as its flagship DSLR. The D5 offers a number of improvements over its predecessor including a new image sensor, new image processor, improved ergonomics and expanded ISO range. Additionally, improved auto focus (AF) modes were introduced. On 23 February 2017, at CP+ show, a special edition was released for Nikon's 100th anniversary.
The Nikon D850 is a professional-grade full-frame digital single-lens reflex camera (DSLR) produced by Nikon. The camera was officially announced on July 25, 2017, launched on August 24, 2017, and first shipped on September 8, 2017. Nikon announced it could not fill the preorders on August 28, 2017 and filled less than 10% of preorders on the first shipping day. It is the successor to the Nikon D810.
The Nikon Z7 is a 45.7 megapixel full-frame mirrorless interchangeable-lens camera produced by Nikon. The camera was officially announced on August 23, 2018, for release in September 2018. It was the first camera to use Nikon's new Z-mount system; the second model, released in November 2018, was the 24.5 megapixel Nikon Z 6.
The Nikon Z6 is a full-frame mirrorless interchangeable-lens camera produced by Nikon. The camera was officially announced on August 23, 2018, to be released in November. Nikon began shipping the Z6 to retailers on November 16, 2018. This was the second camera to use Nikon's new Z-mount system after the release of the 45.75 megapixel Nikon Z 7 in September 2018.