Comparison of Gaussian process software

Last updated

This is a comparison of statistical analysis software that allows doing inference with Gaussian processes often using approximations.

Contents

This article is written from the point of view of Bayesian statistics, which may use a terminology different from the one commonly used in kriging. The next section should clarify the mathematical/computational meaning of the information provided in the table independently of contextual terminology.

Description of columns

This section details the meaning of the columns in the table below.

Solvers

These columns are about the algorithms used to solve the linear system defined by the prior covariance matrix, i.e., the matrix built by evaluating the kernel.

Input

These columns are about the points on which the Gaussian process is evaluated, i.e. if the process is .

Output

These columns are about the values yielded by the process, and how they are connected to the data used in the fit.

Hyperparameters

These columns are about finding values of variables which enter somehow in the definition of the specific problem but that can not be inferred by the Gaussian process fit, for example parameters in the formula of the kernel.

If both the "Prior" and "Posterior" cells contain "Manually", the software provides an interface for computing the marginal likelihood and its gradient w.r.t. hyperparameters, which can be feed into an optimization/sampling algorithm, e.g., gradient descent or Markov chain Monte Carlo.

Linear transformations

These columns are about the possibility of fitting datapoints simultaneously to a process and to linear transformations of it.

Comparison table

Name License Language SolversInputOutput Hyperparameters Linear transformations Name
Exact
Specialized
Approxi­mate
ND
Non-real
LikelihoodErrorsPriorPosterior
Derivative
FiniteSum
PyMC Apache Python YesKroneckerSparseNDNoAnyCorrelatedYesYesNoYesYes PyMC
Stan BSD, GPL customYesNoNoNDNoAnyCorrelatedYesYesNoYesYes Stan
scikit-learn BSD Python YesNoNoNDYesBernoulliUncorrelatedManuallyManuallyNoNoNo scikit-learn
fbm
[7]
Free C YesNoNoNDNoBernoulli, PoissonUncorrelated, StationaryManyYesNoNoYes fbm
GPML
[8] [7]
BSD MATLAB YesNoSparseNDNoManyi.i.d.ManuallyManuallyNoNoNo GPML
GPstuff
[7]
GNU GPL MATLAB, R YesMarkovSparseNDNoManyCorrelatedManyYesFirst RBFNoYes GPstuff
GPy
[9]
BSD Python YesNoSparseNDNoManyUncorrelatedYesYesNoNoNo GPy
GPflow
[9]
Apache Python YesNoSparseNDNoManyUncorrelatedYesYesNoNoNo GPflow
GPyTorch
[10]
MIT Python YesToeplitz, KroneckerSparseNDNoManyUncorrelatedYesYesFirst RBFManuallyManually GPyTorch
GPvecchia
[11]
GNU GPL R YesNoSparse, Hierarch­icalNDNoExponential familyUncorrelatedNoNoNoNoNo GPvecchia
pyGPs
[12]
BSD Python YesNoSparseNDGraphs, ManuallyBernoullii.i.d.ManuallyManuallyNoNoNo pyGPs
gptk
[13]
BSD R YesBlock?SparseNDNoGaussianNoManuallyManuallyNoNoNo gptk
celerite
[3]
MIT Python, Julia, C++ NoSemisep. [a] No1DNoGaussianUncorrelatedManuallyManuallyNoNoNo celerite
george
[6]
MIT Python, C++ YesNoHierarch­icalNDNoGaussianUncorrelatedManuallyManuallyNoNoManually george
neural-tangents
[14] [b]
Apache Python YesBlock, KroneckerNoNDNoGaussianNoNoNoNoNoNo neural-tangents
DiceKriging
[15]
GNU GPL R YesNoNoNDNo?GaussianUncorrelatedSCAD RBFMAPNoNoNo DiceKriging
OpenTURNS
[16]
GNU LGPL Python, C++ YesNoNoNDNoGaussianUncorrelatedManually (no grad.)MAPNoNoNo OpenTURNS
UQLab
[17]
Proprietary MATLAB YesNoNoNDNoGaussianCorrelatedNoMAPNoNoNo UQLab
ooDACE [18] Proprietary MATLAB YesNoNoNDNoGaussianCorrelatedNoMAPNoNoNo ooDACE
DACE Proprietary MATLAB YesNoNoNDNoGaussianNoNoMAPNoNoNo DACE
GpGp MIT R NoNoSparseNDNoGaussiani.i.d.ManuallyManuallyNoNoNo GpGp
SuperGauss GNU GPL R, C++ NoToeplitz [c] No1DNoGaussianNoManuallyManuallyNoNoNo SuperGauss
STK GNU GPL MATLAB YesNoNoNDNoGaussianUncorrelatedManuallyManuallyNoNoManually STK
GSTools GNU LGPL Python YesNoNoNDNoGaussianYesYesYesYesNoNo GSTools
PyKrige BSD Python YesNoNo2D,3DNoGaussiani.i.d.NoNoNoNoNo PyKrige
GPR Apache C++ YesNoSparseNDNoGaussiani.i.d.Some, ManuallyManuallyFirstNoNo GPR
celerite2 MIT Python NoSemisep. [a] No1DNoGaussianUncorrelatedManually [d] ManuallyNoNoYes celerite2
SMT
[19] [20]
BSD Python YesNoSparse, PODI [e] , otherNDNoGaussiani.i.d.SomeSomeFirstNoNo SMT
GPJax Apache Python YesNoSparseNDGraphsBernoulliNoYesYesNoNoNo GPJax
Stheno MIT Python YesLow rankSparseNDNoGaussiani.i.d.ManuallyManuallyApproxi­mateNoYes Stheno
Egobox-gp
[22]
Apache Rust YesNoSparseNDNoGaussiani.i.d.NoMAPFirstNoNo Egobox-gp
Name License Language Exact
Specialized
Approxi­mate
ND
Non-real
LikelihoodErrorsPriorPosterior
Derivative
FiniteSumName
SolversInputOutputHyperparametersLinear transformations

Notes

  1. 1 2 celerite implements only a specific subalgebra of kernels which can be solved in . [3]
  2. neural-tangents is a specialized package for infinitely wide neural networks.
  3. SuperGauss implements a superfast Toeplitz solver with computational complexity .
  4. celerite2 has a PyMC3 interface.
  5. PODI (Proper Orthogonal Decomposition + Interpolation) is an approximation for high-dimensional multioutput regressions. The regression function is lower-dimensional than the outcomes, and the subspace is chosen with the PCA of the (outcome, dependent variable) data. Each principal component is modeled with an a priori independent Gaussian process. [21]

References

  1. P. Cunningham, John; Gilboa, Elad; Saatçi, Yunus (Feb 2015). "Scaling Multidimensional Inference for Structured Gaussian Processes". IEEE Transactions on Pattern Analysis and Machine Intelligence. 37 (2): 424–436. arXiv: 1209.4120 . Bibcode:2015ITPAM..37..424G. doi:10.1109/TPAMI.2013.192. PMID   26353252. S2CID   6878550.
  2. Leith, D. J.; Zhang, Yunong; Leithead, W. E. (2005). "Time-series Gaussian Process Regression Based on Toeplitz Computation of O(N²) Operations and O(N)-level Storage". Proceedings of the 44th IEEE Conference on Decision and Control. pp. 3711–3716. doi:10.1109/CDC.2005.1582739. ISBN   0-7803-9567-0. S2CID   13627455.
  3. 1 2 3 Foreman-Mackey, Daniel; Angus, Ruth; Agol, Eric; Ambikasaran, Sivaram (9 November 2017). "Fast and Scalable Gaussian Process Modeling with Applications to Astronomical Time Series". The Astronomical Journal. 154 (6): 220. arXiv: 1703.09710 . Bibcode:2017AJ....154..220F. doi: 10.3847/1538-3881/aa9332 . S2CID   88521913.
  4. Sarkka, Simo; Solin, Arno; Hartikainen, Jouni (2013). "Spatiotemporal Learning via Infinite-Dimensional Bayesian Filtering and Smoothing: A Look at Gaussian Process Regression Through Kalman Filtering". IEEE Signal Processing Magazine. 30 (4): 51–61. doi:10.1109/MSP.2013.2246292. S2CID   7485363.
  5. Quiñonero-Candela, Joaquin; Rasmussen, Carl Edward (5 December 2005). "A Unifying View of Sparse Approximate Gaussian Process Regression". Journal of Machine Learning Research. 6: 1939–1959. Retrieved 23 May 2020.
  6. 1 2 Ambikasaran, S.; Foreman-Mackey, D.; Greengard, L.; Hogg, D. W.; O’Neil, M. (1 Feb 2016). "Fast Direct Methods for Gaussian Processes". IEEE Transactions on Pattern Analysis and Machine Intelligence. 38 (2): 252–265. arXiv: 1403.6015 . doi:10.1109/TPAMI.2015.2448083. PMID   26761732. S2CID   15206293.
  7. 1 2 3 Vanhatalo, Jarno; Riihimäki, Jaakko; Hartikainen, Jouni; Jylänki, Pasi; Tolvanen, Ville; Vehtari, Aki (Apr 2013). "GPstuff: Bayesian Modeling with Gaussian Processes". Journal of Machine Learning Research. 14: 1175−1179. Retrieved 23 May 2020.
  8. Rasmussen, Carl Edward; Nickisch, Hannes (Nov 2010). "Gaussian processes for machine learning (GPML) toolbox". Journal of Machine Learning Research. 11 (2): 3011–3015. doi:10.1016/0002-9610(74)90157-3. PMID   4204594.
  9. 1 2 Matthews, Alexander G. de G.; van der Wilk, Mark; Nickson, Tom; Fujii, Keisuke; Boukouvalas, Alexis; León-Villagrá, Pablo; Ghahramani, Zoubin; Hensman, James (April 2017). "GPflow: A Gaussian process library using TensorFlow". Journal of Machine Learning Research. 18 (40): 1–6. arXiv: 1610.08733 . Retrieved 6 July 2020.
  10. Gardner, Jacob R; Pleiss, Geoff; Bindel, David; Weinberger, Kilian Q; Wilson, Andrew Gordon (2018). "GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration" (PDF). Advances in Neural Information Processing Systems. 31: 7576–7586. arXiv: 1809.11165 . Retrieved 23 May 2020.
  11. Zilber, Daniel; Katzfuss, Matthias (January 2021). "Vecchia–Laplace approximations of generalized Gaussian processes for big non-Gaussian spatial data". Computational Statistics & Data Analysis. 153 107081. arXiv: 1906.07828 . doi:10.1016/j.csda.2020.107081. ISSN   0167-9473. S2CID   195068888 . Retrieved 1 September 2021.
  12. Neumann, Marion; Huang, Shan; E. Marthaler, Daniel; Kersting, Kristian (2015). "pyGPs — A Python Library for Gaussian Process Regression and Classification". Journal of Machine Learning Research. 16: 2611–2616.
  13. Kalaitzis, Alfredo; Lawrence, Neil D. (May 20, 2011). "A Simple Approach to Ranking Differentially Expressed Gene Expression Time Courses through Gaussian Process Regression". BMC Bioinformatics. 12 (1): 180. doi: 10.1186/1471-2105-12-180 . ISSN   1471-2105. PMC   3116489 . PMID   21599902.
  14. Novak, Roman; Xiao, Lechao; Hron, Jiri; Lee, Jaehoon; Alemi, Alexander A.; Sohl-Dickstein, Jascha; Schoenholz, Samuel S. (2020). "Neural Tangents: Fast and Easy Infinite Neural Networks in Python". International Conference on Learning Representations. arXiv: 1912.02803 .
  15. Roustant, Olivier; Ginsbourger, David; Deville, Yves (2012). "DiceKriging, DiceOptim: Two R Packages for the Analysis of Computer Experiments by Kriging-Based Metamodeling and Optimization". Journal of Statistical Software. 51 (1): 1–55. doi: 10.18637/jss.v051.i01 . S2CID   60672249.
  16. Baudin, Michaël; Dutfoy, Anne; Iooss, Bertrand; Popelin, Anne-Laure (2015). "OpenTURNS: An Industrial Software for Uncertainty Quantification in Simulation". In Roger Ghanem; David Higdon; Houman Owhadi (eds.). Handbook of Uncertainty Quantification. pp. 1–38. arXiv: 1501.05242 . doi:10.1007/978-3-319-11259-6_64-1. ISBN   978-3-319-11259-6. S2CID   88513894.
  17. Marelli, Stefano; Sudret, Bruno (2014). "UQLab: a framework for uncertainty quantification in MATLAB" (PDF). Vulnerability, Uncertainty, and Risk. Quantification, Mitigation, and Management: 2554–2563. doi:10.3929/ethz-a-010238238. ISBN   978-0-7844-1360-9 . Retrieved 28 May 2020.
  18. Couckuyt, Ivo; Dhaene, Tom; Demeester, Piet (2014). "ooDACE toolbox: a flexible object-oriented Kriging implementation" (PDF). Journal of Machine Learning Research. 15: 3183–3186. Retrieved 8 July 2020.
  19. Bouhlel, Mohamed A.; Hwang, John T.; Bartoli, Nathalie; Lafage, Rémi; Morlier, Joseph; Martins, Joaquim R.R.A. (2019). "A Python surrogate modeling framework with derivatives". Advances in Engineering Software. 135 (1): 102662. doi:10.1016/j.advengsoft.2019.03.005.
  20. Saves, Paul; Lafage, Rémi; Bartoli, Nathalie; Diouane, Youssef; Bussemaker, Jasper; Lefebvre, Thierry; Hwang, John T.; Morlier, Joseph; Martins, Joaquim R.R.A. (2024). "SMT 2.0: A Surrogate Modeling Toolbox with a focus on hierarchical and mixed variables Gaussian processes". Advances in Engineering Software. 188 (1): 103571. arXiv: 2305.13998 . doi:10.1016/j.advengsoft.2023.103571.
  21. Porrello, Christian; Dubreuil, Sylvain; Farhat, Charbel (2024). "Bayesian Framework With Projection-Based Model Order Reduction for Efficient Global Optimization". AIAA Aviation Forum and Ascend 2024. p. 4580. doi:10.2514/6.2024-4580. ISBN   978-1-62410-716-0.
  22. Lafage, Rémi (2022). "egobox, a Rust toolbox for efficient global optimization" (PDF). Journal of Open Source Software. 7 (78): 4737. Bibcode:2022JOSS....7.4737L. doi:10.21105/joss.04737.