Efficient computation of limit spectra of sample covariance matrices. Models from random matrix theory (RMT) are increasingly used to gain insights into the behavior of statistical methods under high-dimensional asymptotics. However, the applicability of the framework is limited by numerical problems. Consider the usual model of multivariate statistics where the data is a sample from a multivariate distribution with a given covariance matrix. Under high-dimensional asymptotics, there is a deterministic map from the distribution of eigenvalues of the population covariance matrix (the population spectral distribution or PSD), to the of empirical spectral distribution (ESD). The current methods for computing this map are inefficient, and this limits the applicability of the theory. We propose a new method to compute numerically the ESD from an arbitrary input PSD. Our method, called SPECTRODE, finds the support and the density of the ESD to high precision; we prove this for finite discrete distributions. In computational experiments SPECTRODE outperforms existing methods by orders of magnitude in speed and accuracy. We apply it to compute expectations and contour integrals of the ESD, which are often central in applications.We also illustrate that SPECTRODE is directly useful in statistical problems, such as estimation and hypothesis testing for covariance matrices. Our proposal, implemented in open source software, may broaden the use of RMT in high-dimensional data analysis.
Keywords for this software
References in zbMATH (referenced in 9 articles , 1 standard article )
Showing results 1 to 9 of 9.
- Cordero-Grande, Lucilio: MIXANDMIX: numerical techniques for the computation of empirical spectral distributions of population mixtures (2020)
- Dobriban, Edgar; Leeb, William; Singer, Amit: Optimal prediction in the linearly transformed spiked model (2020)
- Dobriban, Edgar; Owen, Art B.: Deterministic parallel analysis: an improved method for selecting factors and principal components (2019)
- Fan, Zhou; Johnstone, Iain M.: Eigenvalue distributions of variance components estimators in high-dimensional random effects models (2019)
- Heiny, Johannes; Mikosch, Thomas: Almost sure convergence of the largest and smallest eigenvalues of high-dimensional sample correlation matrices (2018)
- Liu, Lydia T.; Dobriban, Edgar; Singer, Amit: (e)PCA: high dimensional exponential family PCA (2018)
- Bun, Joël; Bouchaud, Jean-Philippe; Potters, Marc: Cleaning large correlation matrices: tools from random matrix theory (2017)
- Ledoit, Olivier; Wolf, Michael: Numerical implementation of the QuEST function (2017)
- Dobriban, Edgar: Efficient computation of limit spectra of sample covariance matrices (2014)