On Shrinkage Estimation: Non-orthogonal Case

  • A.K.Md. Ehsanes Saleh School of Mathematics and Statistics, Carleton University, Ottawa, Canada
  • Mina Norouzirad Department of Statistics, Faculty of Mathematical Sciences, Shahrood University of Technology, Shahrood, Iran
DOI: https://doi.org/10.19139/soic.v6i3.582
Keywords: Dominance, Efficiency, LASSO, PTE and Stein-type estimators, Penalty estimator, Risk function, Sparsity.

Abstract

In this paper, we consider the estimation of the parameters of the non-orthogonal regression model, when we suspect a sparsity condition. We provide with a comparative performance characteristics of the primary penalty estimators, namely, the ridge and the LASSO, with the least square estimator, restricted LSE, preliminary test  and Stein-type of  estimators, when the dimension of the parameter  space is less than the dimension  of the sample space. Using the principle of marginal distribution theory, the analysis of risks leads to the following conclusions: (i) ridge estimator outperforms least squares, preliminary test and Stein-type estimators uniformly, (ii) The restricted least squares estimator and LASSO are competitive, although LASSO lags behind the restricted least squares estimator uniformly. Both estimators outperform the least squares, preliminary test, and Stein-type estimators in a subspace, respectively.  (iii) The lower bound risk expression of LASSO does not depend on the threshold parameter. (iv) Performance of the estimators depends upon the size of numbers of active coefficients, non-active coefficients, and the divergence parameter. In support of our conclusion, we prepare some tables and graphs relevant to the properties of the estimators.

References

A. Belloni, and V. Chernozhukov, Least squares after model selection in high-dimensional sparse models, Bernoulli, vol. 2, pp. 521–547, 2013.

L. Breiman, Heuristics of instability and stabilization in model selection, The Annals of Statistics, vol. 24, no. 6, pp. 2350–2383, 1996.

N. R. V. Draper, and R. Craig, Ridge Regression and James- Stein Estimation: Review and Comments, Technometrics, vol. 21, pp. 451–466., 1979.

D. L. Donoho, and I. M. Johnstone, Ideal Spatial Adaptation by Wavelet Shrinkage, Biometrika, vol. 81, no. 3, pp. 425–455, 1994.

J. Fan, and R. Li, Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, vol. 96, no. 456, pp. 1348–1360, 2001.

I. E. Frank, and J. H. Friedman, A Statistical View of Some Chemometrics Regression Tools, Technometrics, vol. 35, no. 2, pp. 109–135, 1993.

B. E. Hansen, The risk of James-Stein and LASSO Shrinkage, Econometric Reviews, vol. 35, no. 8-10, pp. 1456–1470, 2013.

A. E. Hoerl and R. W. Kennard, Ridge regression biased estimation for non-orthogonal problems, Thechnometrics, vol. 12, pp. 69–89, 1970.

G. C. McDonald, and D. I. Galarneau, A Monte Carol evaluation of some ridge-type estimators, J. Amer. Statist. Assoc, vol. 70, pp. 407–416, 1975.

A. K. Md. Ehsanes. Saleh, Theory of Preliminary Test and Stein-Type Estimation with Applications, Wiley; United Stated of America., 2006.

Stamey, T.A., Kabalin, J.N., McNeal, J.E., Johnstone, I.M., Freiha, F., Redwine, E.A. and Yang, N. Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate: II. radical prostatectomy treated patients, Journal of Urology, 141(5), 1076-1083.

R. Tibshirani, Regression shrinkage and selection via the LASSO, J. Royal. Statist. Soc. B., vol. 58, no. 1, pp. 267-288, 1996.

A. N. Tikhonov, Solution of incorrectly formulated problems and the regularization method, Doklady Academia Nauk SSSR, Translated in Soviet Mathematics 4, vol. 151, pp. 501-504, 1963.

H. Zou, The adaptive Lasso and its oracle properties, Journal of the American Statistical Association, vol.101, no. 476, pp. 1418– 1429, 2006.

H. Zou, and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B Stat. Methodol, vol. 67, no. 2, pp. 301-320, 2005.

Published
2018-08-19
How to Cite
Ehsanes Saleh, A., & Norouzirad, M. (2018). On Shrinkage Estimation: Non-orthogonal Case. Statistics, Optimization & Information Computing, 6(3), 427-451. https://doi.org/10.19139/soic.v6i3.582
Issue
Vol 6 No 3 (2018)
Section
Research Articles
Authors who publish with this journal agree to the following terms:

    1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
    2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
    3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).