Zero-inflated model

In statistics, a zero-inflated model is a statistical model based on a zero-inflated probability distribution, i.e. a distribution that allows for frequent zero-valued observations.

Introduction to Zero-Inflated Models

Zero-inflated models are commonly used in the analysis of count data, such as the number of visits a patient makes to the emergency room in one year, or the number of fish caught in one day in one lake. ^[1] Count data can take values of 0, 1, 2, … (non-negative integer values). ^[2] Other examples of count data are the number of hits recorded by a Geiger counter in one minute, patient days in the hospital, goals scored in a soccer game, ^[3] and the number of episodes of hypoglycemia per year for a patient with diabetes. ^[4]

For statistical analysis, the distribution of the counts is often represented using a Poisson distribution or a negative binomial distribution. Hilbe ^[3] notes that "Poisson regression is traditionally conceived of as the basic count model upon which a variety of other count models are based." In a Poisson model, "… the random variable ${\displaystyle y}$ is the count response and parameter ${\displaystyle \lambda }$ (lambda) is the mean. Often, ${\displaystyle \lambda }$ is also called the rate or intensity parameter… In statistical literature, ${\displaystyle \lambda }$ is also expressed as ${\displaystyle \mu }$ (mu) when referring to Poisson and traditional negative binomial models."

In some data, the number of zeros is greater than would be expected using a Poisson distribution or a negative binomial distribution. Data with such an excess of zero counts are described as Zero-inflated. ^[4]

Example histograms of zero-inflated Poisson distributions with mean ${\displaystyle \mu }$ of 5 or 10 and proportion of zero inflation ${\displaystyle \pi }$ of 0.2 or 0.5 are shown below, based on the R program ZeroInflPoiDistPlots.R from Bilder and Laughlin. ^[1]

Examples of Zero-inflated count data

• Fish counts ^[1] "… suppose we recorded the number of fish caught on various lakes in 4-hour fishing trips to Minnesota. Some lakes in Minnesota are too shallow for fish to survive the winter, so fishing in those lakes will yield no catch. On the other hand, even on a lake where fish are plentiful, we may or may not catch any fish due to conditions or our own competence. Thus, the number of fish caught will be zero if the lake does not support fish, and will be zero, one or more if it does."
• Number of wisdom teeth extracted. ^[5] The number of wisdom teeth that a person has had extracted can range from 0 to 4. Some individuals, about one-third of the population, do not have any wisdom teeth. For these individuals, the number of wisdom teeth extracted will always be zero. For other individuals, the number extracted will be between 0 and 4, where a 0 indicates that the subject has not yet, and may never, have any of their 4 wisdom teeth extracted.
• Publications by PhD candidates. ^[6] Long examined the number of publications by 915 doctoral candidates in biochemistry in the last three years of their PhD studies. The proportion of candidates with zero publications exceeded the number predicted by a Poisson model. "Long ^[6] argued that the PhD candidates might fall into two distinct groups: "publishers" (perhaps striving for an academic career) and "non-publishers" (seeking other career paths). One reasonable form of explanation is that the observed zero counts reflect a mixture of the two latent classes – those who simply have not yet published and those who will likely never publish." ^[7]

Zero-inflated data as a mixture of two distributions

As the examples above show, zero-inflated data can arise as a mixture of two distributions. The first distribution generates zeros. The second distribution, which may be a Poisson distribution, a negative binomial distribution or other count distribution, generates counts, some of which may be zeros.". ^[7]

In the statistical literature, different authors may use different names to distinguish zeros from the two distributions. Some authors describe zeros generated by the first (binary) distribution as "structural" and zeros generated by the second (count) distribution as "random". ^[7] Other authors use the terminology "immune" and "susceptible" for the binary and count zeros, respectively ^[1]

Zero-inflated Poisson

Histogram of a zero-inflated Poisson distribution

One well-known zero-inflated model is Diane Lambert's zero-inflated Poisson model, which concerns a random event containing excess zero-count data in unit time. ^[8] For example, the number of insurance claims within a population for a certain type of risk would be zero-inflated by those people who have not taken out insurance against the risk and thus are unable to claim. The zero-inflated Poisson (ZIP) model mixes two zero generating processes. The first process generates zeros. The second process is governed by a Poisson distribution that generates counts, some of which may be zero. The mixture distribution is described as follows:

${\displaystyle \Pr(Y=0)=\pi +(1-\pi )e^{-\lambda }}$

${\displaystyle \Pr(Y=y_{i})=(1-\pi ){\frac {\lambda ^{y_{i}}e^{-\lambda }}{y_{i}!}},\qquad y_{i}=1,2,3,...}$

where the outcome variable ${\displaystyle y_{i}}$ has any non-negative integer value, ${\displaystyle \lambda }$ is the expected Poisson count for the ${\displaystyle i}$th individual; ${\displaystyle \pi }$ is the probability of extra zeros.

The mean is ${\displaystyle (1-\pi )\lambda }$ and the variance is ${\displaystyle \lambda (1-\pi )(1+\pi \lambda )}$.

Estimators of ZIP parameters

The method of moments estimators are given by ^[9]

${\displaystyle {\hat {\lambda }}_{mo}={\frac {s^{2}+m^{2}}{m}}-1,}$

${\displaystyle {\hat {\pi }}_{mo}={\frac {s^{2}-m}{s^{2}+m^{2}-m}},}$

where ${\displaystyle m}$ is the sample mean and ${\displaystyle s^{2}}$ is the sample variance.

The maximum likelihood estimator ^[10] can be found by solving the following equation

${\displaystyle m(1-e^{-{\hat {\lambda }}_{ml}})={\hat {\lambda }}_{ml}\left(1-{\frac {n_{0}}{n}}\right).}$

where ${\displaystyle {\frac {n_{0}}{n}}}$ is the observed proportion of zeros.

A closed form solution of this equation is given by ^[11]

${\displaystyle {\hat {\lambda }}_{ml}=W_{0}(-se^{-s})+s}$

with ${\displaystyle W_{0}}$ being the main branch of Lambert's W-function ^[12] and

${\displaystyle s={\frac {m}{1-{\frac {n_{0}}{n}}}}}$.

Alternatively, the equation can be solved by iteration. ^[13]

The maximum likelihood estimator for ${\displaystyle \pi }$ is given by

${\displaystyle {\hat {\pi }}_{ml}=1-{\frac {m}{{\hat {\lambda }}_{ml}}}.}$

Related models

In 1994, Greene considered the zero-inflated negative binomial (ZINB) model. ^[14] Daniel B. Hall adapted Lambert's methodology to an upper-bounded count situation, thereby obtaining a zero-inflated binomial (ZIB) model. ^[15]

Discrete pseudo compound Poisson model

If the count data ${\displaystyle Y}$ is such that the probability of zero is larger than the probability of nonzero, namely

${\displaystyle \Pr(Y=0)>0.5}$

then the discrete data ${\displaystyle Y}$ obey discrete pseudo compound Poisson distribution. ^[16]

In fact, let ${\displaystyle G(z)=\sum \limits _{n=0}^{\infty }P(Y=n)z^{n}}$ be the probability generating function of ${\displaystyle y_{i}}$. If ${\displaystyle p_{0}=\Pr(Y=0)>0.5}$, then ${\displaystyle |G(z)|\geqslant p_{0}-\sum \limits _{i=1}^{\infty }p_{i}=2p_{0}-1>0}$. Then from the Wiener–Lévy theorem, ^[17] ${\displaystyle G(z)}$ has the probability generating function of the discrete pseudo compound Poisson distribution.

We say that the discrete random variable ${\displaystyle Y}$ satisfying probability generating function characterization

${\displaystyle G_{Y}(z)=\sum \limits _{n=0}^{\infty }P(Y=n)z^{n}=\exp \left(\sum _{k=1}^{\infty }\alpha _{k}\lambda (z^{k}-1)\right),\quad (|z|\leq 1)}$

has a discrete pseudo compound Poisson distribution with parameters

${\displaystyle (\lambda _{1},\lambda _{2},\ldots )=(\alpha _{1}\lambda ,\alpha _{2}\lambda ,\ldots )\in \mathbb {R} ^{\infty }\left(\sum _{k=1}^{\infty }\alpha _{k}=1,\sum \limits _{k=1}^{\infty }|\alpha _{k}|<\infty ,\alpha _{k}\in \mathbb {R} ,\lambda >0\right).}$

When all the ${\displaystyle \alpha _{k}}$ are non-negative, it is the discrete compound Poisson distribution (non-Poisson case) with overdispersion property.

See also

• Poisson distribution
• Zero-truncated Poisson distribution
• Compound Poisson distribution
• Sparse approximation
• Hurdle model

Software

• pscl, glmmTMB and brms R packages

References

1. Bilder, Christopher; Loughin, Thomas (2015), Analysis of Categorical Data with R (First ed.), CRC Press / Chapman & Hall, ISBN   978-1439855676
2. Hilbe, Joseph M. (2014), Modeling Count Data (First ed.), Cambridge University Press, ISBN   978-1107611252
3. Hilbe, Joseph M. (2007), Negative Binomial Regression (Second ed.), Cambridge University Press, ISBN   978-0521198158
4. Lachin, John M. (2011), Biostatistical Methods: The Assessment of Relative Risks (Second ed.), Wiley, ISBN   978-0470508220
5. "Biostatistics II. 1.3 - Zero-inflated Models". YouTube . Retrieved July 1, 2022.
6. Long, J. Scott (1997), Regression Models for Categorical and Limited Dependent Variables (First ed.), Sage Publications, ISBN   978-0803973749
7. Friendly, Michael; David, Thomas (2016), Discrete Data Analysis with R (First ed.), CRC Press / Chapman & Hall, ISBN   978-1498725835
8. Lambert, Diane (1992). "Zero-Inflated Poisson Regression, with an Application to Defects in Manufacturing". Technometrics. 34 (1): 1–14. doi:10.2307/1269547. JSTOR   1269547.
9. Beckett, Sadie; Jee, Joshua; Ncube, Thalepo; Washington, Quintel; Singh, Anshuman; Pal, Nabendu (2014). "Zero-inflated Poisson (ZIP) distribution: parameter estimation and applications to model data from natural calamities". Involve. 7 (6): 751–767. doi: 10.2140/involve.2014.7.751 .
10. Johnson, Norman L.; Kotz, Samuel; Kemp, Adrienne W. (1992). Univariate Discrete Distributions (2nd ed.). Wiley. pp. 312–314. ISBN   978-0-471-54897-3.
11. Dencks, Stefanie; Piepenbrock, Marion; Schmitz, Georg (2020). "Assessing Vessel Reconstruction in Ultrasound Localization Microscopy by Maximum-Likelihood Estimation of a Zero-Inflated Poisson Model". IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control. doi: 10.1109/TUFFC.2020.2980063 .
12. Corless, R. M.; Gonnet, G. H.; Hare, D. E. G.; Jeffrey, D. J.; Knuth, D. E. (1996). "On the Lambert W Function". Advances in Computational Mathematics. 5 (1): 329–359. arXiv: 1809.07369 . doi:10.1007/BF02124750.
13. Böhning, Dankmar; Dietz, Ekkehart; Schlattmann, Peter; Mendonca, Lisette; Kirchner, Ursula (1999). "The zero-inflated Poisson model and the decayed, missing and filled teeth index in dental epidemiology". Journal of the Royal Statistical Society, Series A. 162 (2): 195–209. doi:10.1111/1467-985x.00130.
14. Greene, William H. (1994). "Some Accounting for Excess Zeros and Sample Selection in Poisson and Negative Binomial Regression Models". Working Paper EC-94-10: Department of Economics, New York University. SSRN   1293115.
15. Hall, Daniel B. (2000). "Zero-Inflated Poisson and Binomial Regression with Random Effects: A Case Study". Biometrics. 56 (4): 1030–1039. doi:10.1111/j.0006-341X.2000.01030.x.
16. Huiming, Zhang; Yunxiao Liu; Bo Li (2014). "Notes on discrete compound Poisson model with applications to risk theory". Insurance: Mathematics and Economics. 59: 325–336. doi:10.1016/j.insmatheco.2014.09.012.
17. Zygmund, A. (2002). Trigonometric Series. Cambridge: Cambridge University Press. p. 245.