Empirical dynamic modeling

Empirical dynamic modeling (EDM) is a framework for analysis and prediction of nonlinear dynamical systems. Applications include population dynamics,^{[ improper synthesis? ] [1] [2] [3] [4] [5] [6]} ecosystem service, ^[7] medicine, ^[8] neuroscience, ^{[9] [10] [11]} dynamical systems, ^{[12] [13] [14]} geophysics, ^{[15] [16] [17]} and human-computer interaction. ^[18] EDM was originally developed by Robert May and George Sugihara. It can be considered a methodology for data modeling, predictive analytics, dynamical system analysis, machine learning and time series analysis.^{[ citation needed ]}

Description

Mathematical models have tremendous power to describe observations of real-world systems. They are routinely used to test hypothesis, explain mechanisms and predict future outcomes. However, real-world systems are often nonlinear and multidimensional, in some instances rendering explicit equation-based modeling problematic. Empirical models, which infer patterns and associations from the data instead of using hypothesized equations, represent a natural and flexible framework for modeling complex dynamics.^{[ citation needed ]}

Donald DeAngelis and Simeon Yurek illustrated that canonical statistical models are ill-posed when applied to nonlinear dynamical systems. ^[19] A hallmark of nonlinear dynamics is state-dependence: system states are related to previous states governing transition from one state to another. EDM operates in this space, the multidimensional state-space of system dynamics rather than on one-dimensional observational time series. EDM does not presume relationships among states, for example, a functional dependence, but projects future states from localised, neighboring states. EDM is thus a state-space, nearest-neighbors paradigm where system dynamics are inferred from states derived from observational time series. This provides a model-free representation of the system naturally encompassing nonlinear dynamics.^{[ citation needed ]}

A cornerstone of EDM is recognition that time series observed from a dynamical system can be transformed into higher-dimensional state-spaces by time-delay embedding with Takens's theorem. The state-space models are evaluated based on in-sample fidelity to observations, conventionally with Pearson correlation between predictions and observations.^{[ citation needed ]}

Methods

Primary EDM algorithms include Simplex projection, ^[20] Sequential locally weighted global linear maps (S-Map) projection, ^[21] Multivariate embedding in Simplex or S-Map, ^[1] Convergent cross mapping (CCM), ^[22] and Multiview Embeding, ^[23] described below.

Nomenclature

Parameter	Description
${\displaystyle E}$	embedding dimension
${\displaystyle k}$	number of nearest neighbors
${\displaystyle T_{p}}$	prediction interval
${\displaystyle X\in \mathbb {R} }$	observed time series
${\displaystyle y\in \mathbb {R} ^{E}}$	vector of lagged observations
${\displaystyle \theta \geq 0}$	S-Map localization
${\displaystyle X_{t}^{E}=(X_{t},X_{t-1},\dots ,X_{t-E+1})\in \mathbb {R} ^{E}}$	lagged embedding vectors
${\displaystyle \|v\|}$	norm of v
${\displaystyle N=\{N_{1},\dots ,N_{k}\}}$	list of nearest neighbors

Nearest neighbors are found according to: ${\displaystyle {\text{NN}}(y,X,k)=\|X_{N_{i}}^{E}-y\|\leq \|X_{N_{j}}^{E}-y\|{\text{ if }}1\leq i\leq j\leq k}$

Simplex

Simplex projection ^{[20] [24] [25] [26]} is a nearest neighbor projection. It locates the ${\displaystyle k}$ nearest neighbors to the location in the state-space from which a prediction is desired. To minimize the number of free parameters ${\displaystyle k}$ is typically set to ${\displaystyle E+1}$ defining an ${\displaystyle E+1}$ dimensional simplex in the state-space. The prediction is computed as the average of the weighted phase-space simplex projected ${\displaystyle Tp}$ points ahead. Each neighbor is weighted proportional to their distance to the projection origin vector in the state-space.^{[ citation needed ]}

1. Find ${\displaystyle k}$ nearest neighbor: ${\displaystyle N_{k}\gets {\text{NN}}(y,X,k)}$
2. Define the distance scale: ${\displaystyle d\gets \|X_{N_{1}}^{E}-y\|}$
3. Compute weights: For{${\displaystyle i=1,\dots ,k}$} : ${\displaystyle w_{i}\gets \exp(-\|X_{N_{i}}^{E}-y\|/d)}$
4. Average of state-space simplex: ${\displaystyle {\hat {y}}\gets \sum _{i=1}^{k}\left(w_{i}X_{N_{i}+T_{p}}\right)/\sum _{i=1}^{k}w_{i}}$

S-Map

S-Map ^[21] extends the state-space prediction in Simplex from an average of the ${\displaystyle E+1}$ nearest neighbors to a linear regression fit to all neighbors, but localised with an exponential decay kernel. The exponential localisation function is ${\displaystyle F(\theta )={\text{exp}}(-\theta d/D)}$, where ${\displaystyle d}$ is the neighbor distance and ${\displaystyle D}$ the mean distance. In this way, depending on the value of ${\displaystyle \theta }$, neighbors close to the prediction origin point have a higher weight than those further from it, such that a local linear approximation to the nonlinear system is reasonable. This localisation ability allows one to identify an optimal local scale, in-effect quantifying the degree of state dependence, and hence nonlinearity of the system.^{[ citation needed ]}

Another feature of S-Map is that for a properly fit model, the regression coefficients between variables have been shown to approximate the gradient (directional derivative) of variables along the manifold. ^[27] These Jacobians represent the time-varying interaction strengths between system variables.

1. Find ${\displaystyle k}$ nearest neighbor: ${\displaystyle N\gets {\text{NN}}(y,X,k)}$
2. Sum of distances: ${\displaystyle D\gets {\frac {1}{k}}\sum _{i=1}^{k}\|X_{N_{i}}^{E}-y\|}$
3. Compute weights: For{${\displaystyle i=1,\dots ,k}$} : ${\displaystyle w_{i}\gets \exp(-\theta \|X_{N_{i}}^{E}-y\|/D)}$
4. Reweighting matrix: ${\displaystyle W\gets {\text{diag}}(w_{i})}$
5. Design matrix: ${\displaystyle A\gets {\begin{bmatrix}1&X_{N_{1}}&X_{N_{1}-1}&\dots &X_{N_{1}-E+1}\\1&X_{N_{2}}&X_{N_{2}-1}&\dots &X_{N_{2}-E+1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&X_{N_{k}}&X_{N_{k}-1}&\dots &X_{N_{k}-E+1}\end{bmatrix}}}$
6. Weighted design matrix: ${\displaystyle A\gets WA}$
7. Response vector at ${\displaystyle Tp}$: ${\displaystyle b\gets {\begin{bmatrix}X_{N_{1}+T_{p}}\\X_{N_{2}+T_{p}}\\\vdots \\X_{N_{k}+T_{p}}\end{bmatrix}}}$
8. Weighted response vector: ${\displaystyle b\gets Wb}$
9. Least squares solution (SVD): ${\displaystyle {\hat {c}}\gets {\text{argmin}}_{c}\|Ac-b\|_{2}^{2}}$
10. Local linear model ${\displaystyle {\hat {c}}}$ is prediction: ${\displaystyle {\hat {y}}\gets {\hat {c}}_{0}+\sum _{i=1}^{E}{\hat {c}}_{i}y_{i}}$

Multivariate Embedding

Multivariate Embedding ^{[1] [12] [28]} recognizes that time-delay embeddings are not the only valid state-space construction. In Simplex and S-Map one can generate a state-space from observational vectors, or time-delay embeddings of a single observational time series, or both.

Convergent Cross Mapping

Convergent cross mapping (CCM) ^[22] leverages a corollary to the Generalized Takens Theorem ^[12] that it should be possible to cross predict or cross map between variables observed from the same system. Suppose that in some dynamical system involving variables ${\displaystyle X}$ and ${\displaystyle Y}$, ${\displaystyle X}$ causes ${\displaystyle Y}$. Since ${\displaystyle X}$ and ${\displaystyle Y}$ belong to the same dynamical system, their reconstructions (via embeddings) ${\displaystyle M_{x}}$, and ${\displaystyle M_{y}}$, also map to the same system.

The causal variable ${\displaystyle X}$ leaves a signature on the affected variable ${\displaystyle Y}$, and consequently, the reconstructed states based on ${\displaystyle Y}$ can be used to cross predict values of ${\displaystyle X}$. CCM leverages this property to infer causality by predicting ${\displaystyle X}$ using the ${\displaystyle M_{y}}$ library of points (or vice versa for the other direction of causality), while assessing improvements in cross map predictability as larger and larger random samplings of ${\displaystyle M_{y}}$ are used. If the prediction skill of ${\displaystyle X}$ increases and saturates as the entire ${\displaystyle M_{y}}$ is used, this provides evidence that ${\displaystyle X}$ is casually influencing ${\displaystyle Y}$.

Multiview Embedding

Multiview Embedding ^[23] is a Dimensionality reduction technique where a large number of state-space time series vectors are combitorially assessed towards maximal model predictability.

Extensions

Extensions to EDM techniques include:

• Generalized Theorems for Nonlinear State Space Reconstruction ^[12]
• Extended Convergent Cross Mapping ^[13]
• Dynamic stability ^[4]
• S-Map regularization ^[29]
• Visual analytics with EDM ^[30]
• Convergent Cross Sorting ^[31]
• Expert system with EDM hybrid ^[32]
• Sliding windows based on the extended convergent cross-mapping ^[33]
• Empirical Mode Modeling ^[17]
• Accounting for missing data and variable step sizes ^[34]
• Hierarchical Bayesian EDM via Gaussian processes ^[35]
• Optimal control via Empirical dynamic programming ^[36]
• Multiview distance regularised S-map ^[37]

See also

• System dynamics
• Complex dynamics
• Nonlinear dimensionality reduction

References

1. Dixon, Paul A.; Milicich, Maria J.; Sugihara, George (5 March 1999). "Episodic Fluctuations in Larval Supply". Science. 283 (5407): 1528–1530. Bibcode:1999Sci...283.1528D. doi:10.1126/science.283.5407.1528. PMID   10066174.
2. Ye, Hao; Beamish, Richard J.; Glaser, Sarah M.; Grant, Sue C. H.; Hsieh, Chih-hao; Richards, Laura J.; Schnute, Jon T.; Sugihara, George (31 March 2015). "Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling". Proceedings of the National Academy of Sciences. 112 (13): E1569-76. Bibcode:2015PNAS..112E1569Y. doi: 10.1073/pnas.1417063112 . PMC   4386326 . PMID   25733874.
3. Deyle, Ethan R.; Fogarty, Michael; Hsieh, Chih-hao; Kaufman, Les; MacCall, Alec D.; Munch, Stephan B.; Perretti, Charles T.; Ye, Hao; Sugihara, George (16 April 2013). "Predicting climate effects on Pacific sardine". Proceedings of the National Academy of Sciences. 110 (16): 6430–6435. Bibcode:2013PNAS..110.6430D. doi: 10.1073/pnas.1215506110 . PMC   3631642 . PMID   23536299.
4. Ushio, Masayuki; Hsieh, Chih-hao; Masuda, Reiji; Deyle, Ethan R; Ye, Hao; Chang, Chun-Wei; Sugihara, George; Kondoh, Michio (15 February 2018). "Fluctuating interaction network and time-varying stability of a natural fish community". Nature. 554 (7692): 360–363. Bibcode:2018Natur.554..360U. doi:10.1038/nature25504. PMID   29414940.
5. Deyle, Ethan R.; May, Robert M.; Munch, Stephan B.; Sugihara, George (13 January 2016). "Tracking and forecasting ecosystem interactions in real time". Proceedings of the Royal Society B: Biological Sciences. 283 (1822): 20152258. doi:10.1098/rspb.2015.2258. PMC   4721089 . PMID   26763700.
6. Rogers, Tanya L.; Munch, Stephan B.; Stewart, Simon D.; Palkovacs, Eric P.; Giron-Nava, Alfredo; Matsuzaki, Shin-ichiro S.; Symons, Celia C. (August 2020). "Trophic control changes with season and nutrient loading in lakes". Ecology Letters. 23 (8): 1287–1297. Bibcode:2020EcolL..23.1287R. doi:10.1111/ele.13532. PMC   7384198 . PMID   32476249.
7. Park, Joseph; Saberski, Erik; Stabenau, Erik; Sugihara, George (5 August 2021). "Dynamics of Florida milk production and total phosphate in Lake Okeechobee". PLOS ONE. 16 (8): e0248910. Bibcode:2021PLoSO..1648910P. doi: 10.1371/journal.pone.0248910 . PMC   8341533 . PMID   34351917.
8. Sugihara, G; Allan, W; Sobel, D; Allan, K D (19 March 1996). "Nonlinear control of heart rate variability in human infants". Proceedings of the National Academy of Sciences. 93 (6): 2608–2613. Bibcode:1996PNAS...93.2608S. doi: 10.1073/pnas.93.6.2608 . PMC   39845 . PMID   8637921.
9. McBride, Joseph C.; Zhao, Xiaopeng; Munro, Nancy B.; Jicha, Gregory A.; Schmitt, Frederick A.; Kryscio, Richard J.; Smith, Charles D.; Jiang, Yang (2015). "Sugihara causality analysis of scalp EEG for detection of early Alzheimer's disease". NeuroImage: Clinical. 7: 258–265. doi:10.1016/j.nicl.2014.12.005. PMC   4300018 . PMID   25610788.
10. Tajima, Satohiro; Yanagawa, Toru; Fujii, Naotaka; Toyoizumi, Taro (19 November 2015). "Untangling Brain-Wide Dynamics in Consciousness by Cross-Embedding". PLOS Computational Biology. 11 (11): e1004537. Bibcode:2015PLSCB..11E4537T. doi: 10.1371/journal.pcbi.1004537 . PMC   4652869 . PMID   26584045.
11. Watanakeesuntorn, Wassapon; Takahashi, Keichi; Ichikawa, Kohei; Park, Joseph; Sugihara, George; Takano, Ryousei; Haga, Jason; Pao, Gerald M. (2020). "Massively Parallel Causal Inference of Whole Brain Dynamics at Single Neuron Resolution". 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS). pp. 196–205. arXiv: 2011.11082 . doi:10.1109/ICPADS51040.2020.00035. ISBN   978-1-7281-9074-7.
12. Deyle, Ethan R.; Sugihara, George (31 March 2011). "Generalized Theorems for Nonlinear State Space Reconstruction". PLOS ONE. 6 (3): e18295. Bibcode:2011PLoSO...618295D. doi: 10.1371/journal.pone.0018295 . PMC   3069082 . PMID   21483839.
13. Ye, Hao; Deyle, Ethan R.; Gilarranz, Luis J.; Sugihara, George (5 October 2015). "Distinguishing time-delayed causal interactions using convergent cross mapping". Scientific Reports. 5 (1) 14750. Bibcode:2015NatSR...514750Y. doi:10.1038/srep14750. PMC   4592974 . PMID   26435402.
14. Cenci, Simone; Saavedra, Serguei (29 April 2019). "Non-parametric estimation of the structural stability of non-equilibrium community dynamics". Nature Ecology & Evolution. 3 (6): 912–918. Bibcode:2019NatEE...3..912C. doi:10.1038/s41559-019-0879-1. PMID   31036898.
15. Tsonis, Anastasios A.; Deyle, Ethan R.; May, Robert M.; Sugihara, George; Swanson, Kyle; Verbeten, Joshua D.; Wang, Geli (17 March 2015). "Dynamical evidence for causality between galactic cosmic rays and interannual variation in global temperature". Proceedings of the National Academy of Sciences. 112 (11): 3253–3256. Bibcode:2015PNAS..112.3253T. doi: 10.1073/pnas.1420291112 . PMC   4371914 . PMID   25733877.
16. van Nes, Egbert H.; Scheffer, Marten; Brovkin, Victor; Lenton, Timothy M.; Ye, Hao; Deyle, Ethan; Sugihara, George (May 2015). "Causal feedbacks in climate change". Nature Climate Change. 5 (5): 445–448. Bibcode:2015NatCC...5..445V. doi:10.1038/nclimate2568.
17. Park, Joseph; Pao, Gerald M.; Sugihara, George; Stabenau, Erik; Lorimer, Thomas (May 2022). "Empirical mode modeling: A data-driven approach to recover and forecast nonlinear dynamics from noisy data". Nonlinear Dynamics. 108 (3): 2147–2160. arXiv: 2103.07281 . Bibcode:2022NonDy.108.2147P. doi:10.1007/s11071-022-07311-y.
18. van Berkel, Niels; Dennis, Simon; Zyphur, Michael; Li, Jinjing; Heathcote, Andrew; Kostakos, Vassilis (4 July 2021). "Modeling interaction as a complex system". Human–Computer Interaction. 36 (4): 279–305. doi:10.1080/07370024.2020.1715221. hdl: 11343/247884 .
19. DeAngelis, Donald L.; Yurek, Simeon (31 March 2015). "Equation-free modeling unravels the behavior of complex ecological systems". Proceedings of the National Academy of Sciences. 112 (13): 3856–3857. doi: 10.1073/pnas.1503154112 . PMC   4386356 . PMID   25829536.
20. Sugihara, George; May, Robert M. (April 1990). "Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series". Nature. 344 (6268): 734–741. Bibcode:1990Natur.344..734S. doi:10.1038/344734a0. PMID   2330029.
21. Sugihara, George (15 September 1994). "Nonlinear forecasting for the classification of natural time series". Philosophical Transactions of the Royal Society of London. Series A: Physical and Engineering Sciences. 348 (1688): 477–495. Bibcode:1994RSPTA.348..477S. doi:10.1098/rsta.1994.0106.
22. Sugihara, George; May, Robert; Ye, Hao; Hsieh, Chih-hao; Deyle, Ethan; Fogarty, Michael; Munch, Stephan (26 October 2012). "Detecting Causality in Complex Ecosystems". Science. 338 (6106): 496–500. Bibcode:2012Sci...338..496S. doi:10.1126/science.1227079. PMID   22997134.
23. Ye, Hao; Sugihara, George (26 August 2016). "Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality". Science. 353 (6302): 922–925. Bibcode:2016Sci...353..922Y. doi:10.1126/science.aag0863. PMID   27563095.
24. Takens, Floris (1981). "Detecting strange attractors in turbulence". Dynamical Systems and Turbulence, Warwick 1980. Lecture Notes in Mathematics. Vol. 898. pp. 366–381. doi:10.1007/BFb0091924. ISBN   978-3-540-11171-9.
25. Casdagli, Martin (May 1989). "Nonlinear prediction of chaotic time series". Physica D: Nonlinear Phenomena. 35 (3): 335–356. Bibcode:1989PhyD...35..335C. doi:10.1016/0167-2789(89)90074-2.
26. Judd, Kevin; Mees, Alistair (September 1998). "Embedding as a modeling problem". Physica D: Nonlinear Phenomena. 120 (3–4): 273–286. Bibcode:1998PhyD..120..273J. doi:10.1016/S0167-2789(98)00089-X.
27. Deyle, Ethan R.; May, Robert M.; Munch, Stephan B.; Sugihara, George (13 January 2016). "Tracking and forecasting ecosystem interactions in real time". Proceedings of the Royal Society B: Biological Sciences. 283 (1822): 20152258. doi:10.1098/rspb.2015.2258. PMC   4721089 . PMID   26763700.
28. Sauer, Tim; Yorke, James A.; Casdagli, Martin (November 1991). "Embedology". Journal of Statistical Physics. 65 (3–4): 579–616. Bibcode:1991JSP....65..579S. doi:10.1007/BF01053745.
29. Cenci, Simone; Sugihara, George; Saavedra, Serguei (May 2019). "Regularized S-map for inference and forecasting with noisy ecological time series". Methods in Ecology and Evolution. 10 (5): 650–660. Bibcode:2019MEcEv..10..650C. doi:10.1111/2041-210X.13150.
30. Natsukawa, Hiroaki; Deyle, Ethan R.; Pao, Gerald M.; Koyamada, Koji; Sugihara, George (February 2021). "A Visual Analytics Approach for Ecosystem Dynamics based on Empirical Dynamic Modeling". IEEE Transactions on Visualization and Computer Graphics. 27 (2): 506–516. Bibcode:2021ITVCG..27..506N. doi:10.1109/TVCG.2020.3028956. PMID   33026998.
31. Breston, Leo; Leonardis, Eric J.; Quinn, Laleh K.; Tolston, Michael; Wiles, Janet; Chiba, Andrea A. (13 October 2021). "Convergent cross sorting for estimating dynamic coupling". Scientific Reports. 11 (1) 20374. Bibcode:2021NatSR..1120374B. doi:10.1038/s41598-021-98864-2. PMC   8514556 . PMID   34645847.
32. Deyle, Ethan R.; Bouffard, Damien; Frossard, Victor; Schwefel, Robert; Melack, John; Sugihara, George (2022). "A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures". Proceedings of the National Academy of Sciences. 119 (26) e2102466119. Bibcode:2022PNAS..11902466D. doi: 10.1073/pnas.2102466119 . PMC   9245694 . PMID   35733249.
33. Ge, Xinlei; Lin, Aijing (April 2021). "Dynamic causality analysis using overlapped sliding windows based on the extended convergent cross-mapping". Nonlinear Dynamics. 104 (2): 1753–1765. Bibcode:2021NonDy.104.1753G. doi:10.1007/s11071-021-06362-x.
34. Johnson, Bethany; Munch, Stephan B. (June 2022). "An empirical dynamic modeling framework for missing or irregular samples". Ecological Modelling. 468 109948. Bibcode:2022EcMod.46809948J. doi:10.1016/j.ecolmodel.2022.109948.
35. Munch, Stephan B.; Poynor, Valerie; Arriaza, Juan Lopez (December 2017). "Circumventing structural uncertainty: A Bayesian perspective on nonlinear forecasting for ecology". Ecological Complexity. 32: 134–143. Bibcode:2017EcoCm..32..134M. doi:10.1016/j.ecocom.2016.08.006.
36. Brias, Antoine; Munch, Stephan B. (February 2021). "Ecosystem based multi-species management using Empirical Dynamic Programming". Ecological Modelling. 441 109423. Bibcode:2021EcMod.44109423B. doi:10.1016/j.ecolmodel.2020.109423.
37. Chang, Chun-Wei; Miki, Takeshi; Ushio, Masayuki; Ke, Po-Ju; Lu, Hsiao-Pei; Shiah, Fuh-Kwo; Hsieh, Chih-hao (December 2021). "Reconstructing large interaction networks from empirical time series data". Ecology Letters. 24 (12): 2763–2774. Bibcode:2021EcolL..24.2763C. doi:10.1111/ele.13897. PMID   34601794.

Further reading

• Chang, CW., Ushio, M. & Hsieh, Ch. (2017). "Empirical dynamic modeling for beginners". Ecol Res. 32 (6): 785–796. Bibcode:2017EcoR...32..785C. doi: 10.1007/s11284-017-1469-9 . hdl: 2433/235326 .
• Stephan B Munch, Antoine Brias, George Sugihara, Tanya L Rogers (2020). "Frequently asked questions about nonlinear dynamics and empirical dynamic modelling". ICES Journal of Marine Science. 77 (4): 1463–1479. doi:10.1093/icesjms/fsz209.

External links

Animations

• State Space Reconstruction: Time Series and Dynamic Systems on YouTube
• State Space Reconstruction: Takens' Theorem and Shadow Manifolds on YouTube
• State Space Reconstruction: Convergent Cross Mapping on YouTube

Online books or lecture notes

• EDM Introduction. Introduction with video, examples and references.
• Berglund, Nils (2001). Geometrical theory of dynamical systems (Preprint). arXiv: math.HO/0111177 . Bibcode:2001math.....11177B.
• Arxiv preprint server has daily submissions of (non-refereed) manuscripts in dynamical systems.

Research groups

• Sugihara Lab, Scripps Institution of Oceanography, University of California San Diego.
• Biological Nonlinear Dynamics Data Science Unit, Okinawa Institute of Science and Technology Graduate University, Okinawa Japan.