Retinomorphic sensor

Top: Step change in light intensity incident on sensor as a function of time. Middle: Form of output from conventional optical sensor as a function of time. Bottom: Form of output from retinomorphic sensor as a function of time.

Retinomorphic sensors are a type of event-driven optical sensor which produce a signal in response to changes in light intensity, rather than to light intensity itself. ^[1] This is in contrast to conventional optical sensors such as charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) based sensors, which output a signal that increases with increasing light intensity. Because they respond to movement only, retinomorphic sensors are hoped to enable faster tracking of moving objects than conventional image sensors, and have potential applications in autonomous vehicles, robotics, and neuromorphic engineering. ^{[2] [3] [4] [5]}

Naming and history

The first so-called artificial retina were reported in the late 1980's by Carver Mead and his doctoral students Misha Mahowald, and Tobias Delbrück. ^{[6] [7]} These silicon-based sensors were based on small circuits involving differential amplifiers, capacitors, and resistors. The sensors produced a spike and subsequent decay in output voltage in response to a step-change in illumination intensity. This response is analogous to that of animal retinal cells, which in the 1920's were observed to fire more frequently when the intensity of light was changed than when it was constant. ^[8] The name silicon retina has hence been used to describe these sensors. ^[9]

The term retinomorphic was first used in a conference paper by Lex Akers in 1990. ^[10] The term received wider use by Stanford Professor of Engineering Kwabena Boahen, and has since been applied to a wide range of event-driven sensing strategies. ^[11] The word is analogous to neuromorphic, which is applied to hardware elements (such as processors) designed to replicate the way the brain processes information.

Operating principles

Illustration of charge on the plates of a photosensitive capacitor in series with a resistor. Before time t₀, the capacitor is in the dark and the charge on the plates is determined by C_dark. At time t ≥ t₀, the capacitor is under illumination and the capacitance changes to C_light, resulting in the charge the plates can accommodate changing. The excess charge then flows on/off of the plates over a period of time determined by the resistor R.

There are several retinomorphic sensor designs which yield a similar response. The first designs employed a differential amplifier which compared the input signal from of a conventional sensor (e.g. a phototransistor) to a filtered version of the output, ^[6] resulting in a gradual decay if the input was constant. Since the 1980's these sensors have evolved into much more complex and robust circuits. ^[1]

A more compact design of retinomorphic sensor consists of just a photosensitive capacitor and a resistor in series. ^[12] The output voltage of these retinomorphic sensors, ${\displaystyle V_{out}}$, is defined as voltage dropped across the resistor. The photosensitive capacitor is designed to have a capacitance which is a function of incident light intensity. If a constant voltage ${\displaystyle V_{in}}$, is applied across this RC circuit it will act as a passive high-pass filter and all voltage will be dropped across the capacitor (i.e. ${\displaystyle V_{out}=0}$). After a sufficient amount of time, the plates of the capacitor will be fully charged with a charge ${\displaystyle Q=\pm C_{dark}(V_{in}-V_{out})}$ on each plate, where ${\displaystyle C_{dark}}$ is the capacitance in the dark. Since ${\displaystyle V_{out}=0}$ under constant illumination, this can be simplified to ${\displaystyle Q=\pm C_{dark}V_{in}}$.

If light is then applied to the capacitor it will change capacitance to a new value: ${\displaystyle C_{light}}$. The charge that the plates can accommodate will therefore change to ${\displaystyle Q=\pm C_{light}(V_{in}-V_{out})}$, leaving a surplus / deficit of charge on each plate. The excess charge will be forced to leave the plates, flowing either to ground or the input voltage terminal. The rate of charge flow is determined by the resistance of the resistor ${\displaystyle R}$, and the capacitance of the capacitor. This charge flow will lead to a non-zero voltage being dropped across the resistor and hence a non-zero ${\displaystyle V_{out}}$. After the charge stops flowing the system returns to steady-state, all the voltage is once again dropped across the capacitor, and ${\displaystyle V_{out}=0}$ again.

Left: schematic cross-sectional diagram of photosensitive capacitor. Center: circuit diagram of retinomorphic sensor, with photosensitive capacitor at top. Right: Expected transient response of retinomorphic sensor to application of constant illumination.

For a capacitor to change its capacitance under illumination, the dielectric constant of the insulator between the plates, ^[13] or the effective dimensions of the capacitor, must be illumination-dependent. The effective dimensions can be changed by using a bilayer material between the plates, consisting of an insulator and a semiconductor. Under appropriate illumination conditions the semiconductor will increase its conductivity when exposed to light, emulating the process of moving the plates of the capacitor closer together, and therefore increasing capacitance. For this to be possible, the semiconductor must have a low electrical conductivity in the dark, and have an appropriate band gap to enable charge generation under illumination. The device must also allow optical access to the semiconductor, through a transparent plate (e.g. using a transparent conducting oxide).

Applications

Main article: Event camera

Conventional cameras capture every part of an image, regardless of whether it is relevant to the task. Because every pixel is measured, conventional image sensors are only able to sample the visual field at relatively low frame rates, typically 30 - 240 frames per second. Even in professional high speed cameras used for motion picture, the frame rate is limited to a few 10's of thousands of frames per second for a full resolution image. This limitation could represent a performance bottleneck in the identification of high speed moving objects. This is particularly critical in applications where rapid identification of movement is critical, such as in autonomous vehicles.

By contrast, retinomorphic sensors identify movement by design. This means that they do not have a frame rate and instead are event-driven, responding only when needed. For this reason, retinomorphic sensors are hoped to enable identification of moving objects much more quickly than conventional real-time image analysis strategies. ^[4] Retinomorphic sensors are therefore hoped to have applications in autonomous vehicles, ^{[14] [15]} robotics, ^[16] and neuromorphic engineering. ^[17]

Theory

Retinomorphic sensor operation can be quantified using similar techniques to simple RC circuits, the only difference being that capacitance is not constant as a function of time in a retinomorphic sensor. ^[18] If the input voltage is defined as ${\displaystyle V_{in}}$, the voltage dropped across the resistor as ${\displaystyle V_{out}}$, and the voltage dropped across the capacitor as ${\displaystyle V_{C}}$, we can use Kirchhoff's Voltage Law to state:

${\displaystyle V_{in}=V_{C}+V_{out}}$

Defining the current flowing through the resistor as ${\displaystyle I}$, we can use Ohm's Law to write:

${\displaystyle V_{in}=V_{C}+IR}$

From the definition of current, we can then write this in terms of charge, ${\displaystyle Q}$, flowing off the bottom plate:

${\displaystyle V_{in}=V_{C}+R{\frac {dQ}{dt}}}$

where ${\displaystyle t}$ is time. Charge on the capacitor plates is defined by the product of capacitance, ${\displaystyle C}$, and the voltage across the capacitor, ${\displaystyle V_{C}}$, we can hence say:

${\displaystyle V_{in}=V_{C}+R{\frac {d}{dt}}(CV_{C})}$

Because capacitance in retinomorphic sensors is a function of time, ${\displaystyle C}$ cannot be taken out of the derivative as a constant. Using the product rule, we get the following general equation of retinomorphic sensor response:

${\displaystyle V_{in}=V_{C}+R{\bigg [}C{\frac {dV_{C}}{dt}}+V_{C}{\frac {dC}{dt}}{\bigg ]}}$

or, in terms of the output voltage:

${\displaystyle V_{out}=R{\bigg [}C{\frac {dV_{C}}{dt}}+V_{C}{\frac {dC}{dt}}{\bigg ]}}$

Response to a step-change in intensity

While the equation above is valid for any form of ${\displaystyle C(t)}$, it cannot be solved analytically unless the input form of the optical stimulus is known. The simplest form of optical stimulus would be a step function going from zero to some finite optical power density ${\displaystyle P}$ at a time ${\displaystyle t=t_{0}}$. While real-world applications of retinomorphic sensors are unlikely to be accurately described by such events, it is a useful way to understand and benchmark the performance of retinomorphic sensors. In particular, we are primarily concerned with the maximum height of the ${\displaystyle V_{out}}$ immediately after the light has been turned on.

In this case the capacitance could be described by:

${\displaystyle C(t)={\begin{cases}C_{dark},&{\text{for }}t<t_{0}\\C_{light},&{\text{for }}t\geq t_{0}\end{cases}}}$

The capacitance under illumination will depend on ${\displaystyle P}$. Semiconductors are known ^[19] to have a conductance, ${\displaystyle G}$, which increases with a power-law dependence on incident optical power density: ${\displaystyle G\propto P^{\gamma }}$, where ${\displaystyle \gamma }$ is a dimensionless exponent. Since ${\displaystyle G}$ is linearly proportional to charge density, and capacitance is linearly proportional to charges on the plates for a given voltage, the capacitance of a retinomorphic sensor also has a power-law dependence on ${\displaystyle P}$. The capacitance as a function of time in response to a step function, can therefore be written as:

${\displaystyle C(t)={\begin{cases}C_{dark},&{\text{for }}t<t_{0}\\C_{dark}+\alpha _{0}P^{\gamma },&{\text{for }}t\geq t_{0}\end{cases}}}$

where ${\displaystyle \alpha _{0}}$ is the capacitance prefactor. For a step function we can re-write our differential equation for ${\displaystyle V_{in}}$ as a difference equation:

${\displaystyle V_{in}=V_{C}+R{\bigg [}C{\frac {\Delta V_{C}}{\Delta t}}+V_{C}{\frac {\Delta C}{\Delta t}}{\bigg ]}}$

where ${\displaystyle \Delta V_{C}}$ is the change in voltage dropped across the capacitor as a result of turning on the light, ${\displaystyle \Delta C}$ is the change in capacitance as a result of turning on the light, and ${\displaystyle \Delta t}$ is the time taken for the light to turn on. The variables ${\displaystyle V_{C}}$ and ${\displaystyle C}$ are defined as the voltage dropped across the capacitor and the capacitance, respectively, immediately after the light has been turned on. I.e. ${\displaystyle V_{C}}$ is henceforth shorthand for ${\displaystyle V_{C}(t_{0})}$, and ${\displaystyle C}$ is henceforth shorthand for ${\displaystyle C(t_{0})}$. Assuming the sensor has been held in the dark for sufficiently long before the light is turned on, the change in ${\displaystyle V_{C}}$ can hence be written as:

${\displaystyle \Delta V_{C}=V_{C}-V_{in}}$

Similarly, the change in ${\displaystyle C}$ can be written as

${\displaystyle \Delta C=\alpha _{0}P^{\gamma }}$

Putting these into the difference equation for ${\displaystyle V_{in}}$:

${\displaystyle V_{in}=V_{C}+R{\bigg [}C{\frac {V_{C}-V_{in}}{\Delta t}}+V_{C}{\frac {\alpha _{0}P^{\gamma }}{\Delta t}}{\bigg ]}}$

Top: Illustration of transient response (output voltage ${\displaystyle V_{out}}$ as a function of ${\displaystyle t}$) of retinomorphic sensor in response to a step change in light intensity from 0 to ${\displaystyle P}$, for increasing values of ${\displaystyle P}$. Bottom: Illustration of how one can extract the retinomorphic figure of merit, ${\displaystyle \Lambda }$, from experimental retinomorphic sensor data. By plotting the ratio of ${\displaystyle V_{in}/V_{max}}$ on the y-axis against ${\displaystyle 1/{\sqrt {P}}}$ on the x-axis, then fitting with a straight line, the gradient will yield ${\displaystyle 1/\Lambda }$.

Multiplying this out:

${\displaystyle V_{in}\Delta t=V_{C}\Delta t+R{\bigg [}C(V_{C}-V_{in})+V_{C}\alpha _{0}P^{\gamma }{\bigg ]}}$

Since we are assuming the light turns on very quickly we can approximate ${\displaystyle \Delta t\approx 0}$. This leads to the following:

${\displaystyle V_{C}=V_{in}{\frac {C_{dark}+\alpha _{0}P^{\gamma }}{C_{dark}+2\alpha _{0}P^{\gamma }}}}$

Using the relationship ${\displaystyle V_{in}=V_{C}+V_{out}}$, this can then be written in terms of the output voltage:

${\displaystyle V_{max}=V_{in}{\frac {\alpha _{0}P^{\gamma }}{C_{dark}+2\alpha _{0}P^{\gamma }}}}$

Where we have defined the peak height as ${\displaystyle V_{max}=V_{out}(t_{0})}$, since he peak occurs immediately after the light has been turned on.

The retinomorphic figure of merit, ${\displaystyle \Lambda }$, is defined as the ratio of the capacitance prefactor and the capacitance of the retinomorphic sensor in the dark: ^[18]

${\displaystyle \Lambda ={\frac {\alpha _{0}}{C_{dark}}}}$

With this parameter, the inverse ratio of peak height to input voltage can be written as follows:

${\displaystyle {\frac {V_{in}}{V_{max}}}=2+{\frac {1}{\Lambda P^{\gamma }}}}$

The value of ${\displaystyle \gamma }$ will depend on the nature of recombination in the semiconductor, ^[20] but if band-to-band recombination dominates and the charge density of electrons and holes are equal, ${\displaystyle \gamma =0.5}$. For systems where this is approximately true ^[21] the following simplification to the above equation an be made:

${\displaystyle {\frac {V_{in}}{V_{max}}}=2+{\frac {1}{\Lambda P^{1/2}}}}$

This equation provides a simple method for evaluating the retinomorphic figure of merit from experimental data. This can be carried out by measuring the peak height, ${\displaystyle V_{max}}$, of a retinomorphic sensor in response to a step change in light intensity from 0 to ${\displaystyle P}$, for a range of values ${\displaystyle P}$. Plotting ${\displaystyle V_{in}/V_{max}}$ as a function of ${\displaystyle P^{-1/2}}$ should yield a straight line with a gradient of ${\displaystyle 1/\Lambda }$. This approach assumes that ${\displaystyle V_{max}}$ is linearly proportional to ${\displaystyle V_{in}}$.

See also

• Active-pixel sensor
• Charge coupled device
• Event camera
• Neuromorphic engineering
• Optical sensor
• Photodiode

References

1. Posch, Christoph; Serrano-Gotarredona, Teresa; Linares-Barranco, Bernabe; Delbruck, Tobi (2014). "Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output". Proceedings of the IEEE. 102 (10): 1470–1484. doi:10.1109/JPROC.2014.2346153. hdl: 11441/102353 . ISSN   1558-2256. S2CID   11513955.
2. Mahowald, Misha A.; Mead, Carver (1991). "The Silicon Retina". Scientific American. 264 (5): 76–82. Bibcode:1991SciAm.264e..76M. doi:10.1038/scientificamerican0591-76. PMID   2052936 . Retrieved 2021-12-28.
3. Gilder, George F. (2005). The Silicon Eye. W. W. Norton & Company. ISBN   978-0-393-05763-8.
4. Hambling, David. "AI vision could be improved with sensors that mimic human eyes". New Scientist. Retrieved 2021-10-28.
5. "An eye for an AI: Optic device mimics human retina". BBC Science Focus Magazine. Retrieved 2021-10-28.
6. Delbrück, T.; Mead, C. A. (1989), Touretzky, D. S. (ed.), An Electronic Photoreceptor Sensitive to Small Changes in Intensity, vol. 1, San Mateo, CA: Morgan Kaufmann Publishers, pp. 720–727, ISBN   978-1-55860-015-7 , retrieved 2021-12-23
7. Mead, Carver A.; Mahowald, M. A. (1988-01-01). "A silicon model of early visual processing". Neural Networks. 1 (1): 91–97. doi:10.1016/0893-6080(88)90024-X. ISSN   0893-6080.
8. Adrian, E. D.; Matthews, Rachel (1927). "The action of light on the eye". The Journal of Physiology. 63 (4): 378–414. doi:10.1113/jphysiol.1927.sp002410. ISSN   1469-7793. PMC   1514941 . PMID   16993896.
9. Delbruck, T. (1993). "Silicon retina with correlation-based, velocity-tuned pixels". IEEE Transactions on Neural Networks. 4 (3): 529–541. doi:10.1109/72.217194. ISSN   1941-0093. PMID   18267755.
10. Rao, A.; Akers, L.A. (1990). "A retinomorphic VLSI smart sensor for invariant geometric object recognition". 1990 IJCNN International Joint Conference on Neural Networks. pp. 949–954 vol.2. doi:10.1109/IJCNN.1990.137961. S2CID   35554142.
11. Boahen, K. (1996). "Retinomorphic vision systems". Proceedings of Fifth International Conference on Microelectronics for Neural Networks. pp. 2–14. doi:10.1109/MNNFS.1996.493766. ISBN   0-8186-7373-7. S2CID   62609792.
12. Trujillo Herrera, Cinthya; Labram, John G. (2020-12-07). "A perovskite retinomorphic sensor". Applied Physics Letters. 117 (23): 233501. Bibcode:2020ApPhL.117w3501T. doi: 10.1063/5.0030097 . ISSN   0003-6951. S2CID   230546095.
13. Dillman, Norman (1965). "Photodielectric effect in semiconductors". PhD Thesis.
14. IOM3. "Optical sensor mimics human eye for self-driving cars". www.iom3.org. Retrieved 2021-12-29.
15. "Simple Eyelike Sensors Could Make AI Systems More Efficient". Inside Science. 8 December 2020. Retrieved 2021-12-29.
16. "Bigger boost in robot's field of view". ZME Science. 2021-01-05. Retrieved 2021-12-29.
17. "Perovskite sensor sees more like the human eye". Physics World. 2021-01-18. Retrieved 2021-12-29.
18. Trujillo Herrera, Cinthya; Labram, John G (2021-09-16). "Quantifying the performance of perovskite retinomorphic sensors". Journal of Physics D: Applied Physics. 54 (47): 475110. doi:10.1088/1361-6463/ac1d10. ISSN   0022-3727. S2CID   237541793.
19. "Photoelectronic Properties of Semiconductors | Materials science". Cambridge University Press. Retrieved 2021-12-28.
20. Balberg, I. (1994-01-15). "The two carriers' mobility‐lifetime products and their light intensity dependencies in hydrogenated amorphous silicon". Journal of Applied Physics. 75 (2): 914–923. Bibcode:1994JAP....75..914B. doi:10.1063/1.356447. ISSN   0021-8979.
21. Levine, Igal; Gupta, Satyajit; Brenner, Thomas M.; Azulay, Doron; Millo, Oded; Hodes, Gary; Cahen, David; Balberg, Isaac (2016-12-15). "Mobility–Lifetime Products in MAPbI3 Films". The Journal of Physical Chemistry Letters. 7 (24): 5219–5226. doi:10.1021/acs.jpclett.6b02287. PMID   27973905.