On the inference of entropy measures under different sampling schemes
Keywords:
Burr XII; Shannon, Havrda-Charvat; Tsallis; Rényi Entropy; Arimoto; Ranked Set Sampling.
Abstract
Entropy measures are fundamental measures for quantifying the uncertainty of random variables. In this study, we examine the maximum likelihood estimators (MLE) of five well-known entropy measures: Shannon, Rényi, Havrda, Arimoto, and Tsallis, under both Simple Random Sampling (SRS) and Ranked Set Sampling (RSS). We derived the asymptotic bias and variance for these entropy estimators and conducted extensive simulations to assess the performance of SRS and RSS in estimating these entropy measures. The effectiveness of our estimators was demonstrated using breast cancer data.References
Al-Babtain, A. A., Elbatal, I., Chesneau, C., & Elgarhy, M. (2021). Estimation of different
types of entropies for the Kumaraswamy distribution. PLoS One, 16(3), e0249027.
Al-Hussaini, E. K., & Jaheen, Z. F. (1992). Bayesian estimation of the parameters, reliability
and failure rate functions of the Burr type XII failure model. Journal of statistical c omputation and simulation, 41(1-2), 31-40.
Al-Saleh, M. F. and Al-Kadiri, M. A. (2000). Double ranked set sampling. Statistics and
probability letters, 48(2), 205-212.
Arimoto, S. (1971). Information-theoretical considerations on estimation problems.
Information and control, 19(3), 181-194.
Bantan, R. A., Elgarhy, M., Chesneau, C., & Jamal, F. (2020). Estimation of entropy for
inverse Lomax distribution under multiple censored data. Entropy, 22(6), 601.
Casella, G., & Berger, R. (2024). Statistical inference. CRC Press.
Cho, Y., Sun, H., & Lee, K. (2014). An estimation of the entropy for a Rayleigh distribution
based on doubly-generalized Type-II hybrid censored samples. Entropy, 16(7), 3655-3669.
Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
Dong, G., Shakhatreh, M. K., & He, D. (2024). Bayesian analysis for the Shannon entropy
of the Lomax distribution using noninformative priors. Journal of Statistical Computation and Simulation, 94(6), 1317-1338.
Eftekharian, A., Samawi H., and Amiri, M. (2023). On Kernel-based Estimation of Distribution
Function and its Quantities Based on Ranked Set Sampling. First online in the Journal of Statistical Computation and Simulation (GSCS). 93(11), 1772-1798.
Golan, A. (2008). Information and entropy econometrics: A review and synthesis.
Foundations and trends in econometrics, 2(1–2), 1-145.
Havrda, J., & Charv´at, F. (1967). Quantification method of classification processes.
Concept of structural $ a $-entropy. Kybernetika, 3(1), 30-35.
Jozani, M.J. and Ahmadi, J (2014). On uncertainty and information properties
of ranked set samples. Information Sciences, 264, 291-301.
Kaur, A., Patil, G. P., Sinha, A. K. and Tailie, C. (1995). Ranked set sampling: an
annotated bibliography. Environmental and Ecological Statistics, 2, 25-54.
Kotb, M. S., Newer, H. A., & Mohie El-Din, M. M. (2024). Bayesian Inference for the
Entropy of the Rayleigh Model Based on Ordered Ranked Set Sampling. Annals of Data Science, 1-24.
Kumar, K., Kumar, I., & Ng, H. K. T. (2024). On Estimation of Shannon’s Entropy of
Maxwell Distribution Based on Progressively First- Failure Censored Data. Stats, 7(1), 1 38-159.
Madukaife, M. S., & Phuc, H. D. (2024). Estimation of Shannon differential entropy: An
extensive comparative review. arXiv preprint arXiv:2406.19432.
McIntyre, G. A. (1952). A method for unbiased selective sampling using ranked set.
Australian Journal of Agricultural Research, 3, 385-390.
Muttlak, H. A. (1997). Median ranked set sampling. J. of App. Stat. Sci. 6, 4, 245-255.
Namdari, A., & Li, Z. (2019). A review of entropy measures for uncertainty quantification
of stochastic processes. Advances in Mechanical Engineering, 11(6), 687814019857350.
Patil, G. P., Sinha, A. K. and Taillie, C. (1999). Ranked set sampling: A bibliography
Environ. Ecolog. Statist. 6, 91-98.
Principe, J. C., Xu, D., & Erdogmuns, D. (2010). Renyi’s entropy, divergence and their
nonparametric estimators. Information Theoretic Learning: Renyi's Entropy and Kernel Perspectives, 47-102.
R´enyi, A. (1961, January). On measures of entropy and information. In Proceedings of the
Fourth Berkeley symposium on mathematical statistics and probability, volume 1:
contributions to the theory of statistics (Vol. 4, pp. 547-562). University of California Press.
Robinson, D. W. (2008). Entropy and uncertainty. Entropy, 10(4), 493-506.
Samawi, H. M. Ahmed, M. S. and Abu Dayyeh, W. (1996). Estimating the population
mean using extreme ranked set sampling. Biom. J. 38 (5), 577-586.
Samawi, H. M. and Al-Sageer, O. A. (2001). On the estimation of the distribution
function using extreme and median ranked set sampling. Biom. J. 43(3), 357-373.
Shrahili, M., El-Saeed, A. R., Hassan, A. S., Elbatal, I., & Elgarhy, M. (2022). Estimation of
Entropy for Log-Logistic Distribution under Progressive Type II Censoring. Journal of
Nanomaterials, 2022(1), 2739606.
Sigmon, D., & Fatima, S. (2022). Fine Needle Aspiration. Treasure Island (FL): StatPearls
Publishing.
Singh, V. P. (2013). Entropy theory and its application in environmental and water
engineering. John Wiley & Sons.
Soliman, A. A., Abou-Elheggag, N. A., Abd ellah, A. H., & Modhesh, A. A. (2012).
Bayesian and non-Bayesian inferences of the Burr-XII distribution for progressive first-failure censored data. Metron, 70, 1-25.
Tahmasebi, S., Longobardi , M., Kazemi, M.R. and Alizadeh, M. (2020). Cumulative Tsallis
entropy for maximum ranked set sampling with unequal samples. Physica A: Statistical Mechanics and its Applications, 556, 124763.
Tsallis, C. (1988). Possible generalization of Boltzmann-Gibbs statistics. Journal of
statistical physics, 52, 479-487.
Wang, X., & Gui, W. (2021). Bayesian estimation of entropy for burr type xii distribution
under progressive type-ii censored data. Mathematics, 9(4), 313.
Yu, J., Gui, W., & Shan, Y. (2019). Statistical inference on the Shannon entropy of inverse
Weibull distribution under the progressive first-failure censoring. Entropy, 21(12), 1209.
Zamanzade, E. Mahdizadeh, M. (2017) Entropy Estimation From Ranked Set Samples
With Application to Test of Fit. Revista Colombiana de Estadística, 40(2), 223-241.
Zamanzade, E., Mahdizadeh, M. and Samawi, H. (2024). Nonparametric estimation of mean
residual lifetime in ranked set sampling with a concomitant variable. Journal of Applied Statistics, DOI: 10.1080/02664763.2023.2301334
types of entropies for the Kumaraswamy distribution. PLoS One, 16(3), e0249027.
Al-Hussaini, E. K., & Jaheen, Z. F. (1992). Bayesian estimation of the parameters, reliability
and failure rate functions of the Burr type XII failure model. Journal of statistical c omputation and simulation, 41(1-2), 31-40.
Al-Saleh, M. F. and Al-Kadiri, M. A. (2000). Double ranked set sampling. Statistics and
probability letters, 48(2), 205-212.
Arimoto, S. (1971). Information-theoretical considerations on estimation problems.
Information and control, 19(3), 181-194.
Bantan, R. A., Elgarhy, M., Chesneau, C., & Jamal, F. (2020). Estimation of entropy for
inverse Lomax distribution under multiple censored data. Entropy, 22(6), 601.
Casella, G., & Berger, R. (2024). Statistical inference. CRC Press.
Cho, Y., Sun, H., & Lee, K. (2014). An estimation of the entropy for a Rayleigh distribution
based on doubly-generalized Type-II hybrid censored samples. Entropy, 16(7), 3655-3669.
Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
Dong, G., Shakhatreh, M. K., & He, D. (2024). Bayesian analysis for the Shannon entropy
of the Lomax distribution using noninformative priors. Journal of Statistical Computation and Simulation, 94(6), 1317-1338.
Eftekharian, A., Samawi H., and Amiri, M. (2023). On Kernel-based Estimation of Distribution
Function and its Quantities Based on Ranked Set Sampling. First online in the Journal of Statistical Computation and Simulation (GSCS). 93(11), 1772-1798.
Golan, A. (2008). Information and entropy econometrics: A review and synthesis.
Foundations and trends in econometrics, 2(1–2), 1-145.
Havrda, J., & Charv´at, F. (1967). Quantification method of classification processes.
Concept of structural $ a $-entropy. Kybernetika, 3(1), 30-35.
Jozani, M.J. and Ahmadi, J (2014). On uncertainty and information properties
of ranked set samples. Information Sciences, 264, 291-301.
Kaur, A., Patil, G. P., Sinha, A. K. and Tailie, C. (1995). Ranked set sampling: an
annotated bibliography. Environmental and Ecological Statistics, 2, 25-54.
Kotb, M. S., Newer, H. A., & Mohie El-Din, M. M. (2024). Bayesian Inference for the
Entropy of the Rayleigh Model Based on Ordered Ranked Set Sampling. Annals of Data Science, 1-24.
Kumar, K., Kumar, I., & Ng, H. K. T. (2024). On Estimation of Shannon’s Entropy of
Maxwell Distribution Based on Progressively First- Failure Censored Data. Stats, 7(1), 1 38-159.
Madukaife, M. S., & Phuc, H. D. (2024). Estimation of Shannon differential entropy: An
extensive comparative review. arXiv preprint arXiv:2406.19432.
McIntyre, G. A. (1952). A method for unbiased selective sampling using ranked set.
Australian Journal of Agricultural Research, 3, 385-390.
Muttlak, H. A. (1997). Median ranked set sampling. J. of App. Stat. Sci. 6, 4, 245-255.
Namdari, A., & Li, Z. (2019). A review of entropy measures for uncertainty quantification
of stochastic processes. Advances in Mechanical Engineering, 11(6), 687814019857350.
Patil, G. P., Sinha, A. K. and Taillie, C. (1999). Ranked set sampling: A bibliography
Environ. Ecolog. Statist. 6, 91-98.
Principe, J. C., Xu, D., & Erdogmuns, D. (2010). Renyi’s entropy, divergence and their
nonparametric estimators. Information Theoretic Learning: Renyi's Entropy and Kernel Perspectives, 47-102.
R´enyi, A. (1961, January). On measures of entropy and information. In Proceedings of the
Fourth Berkeley symposium on mathematical statistics and probability, volume 1:
contributions to the theory of statistics (Vol. 4, pp. 547-562). University of California Press.
Robinson, D. W. (2008). Entropy and uncertainty. Entropy, 10(4), 493-506.
Samawi, H. M. Ahmed, M. S. and Abu Dayyeh, W. (1996). Estimating the population
mean using extreme ranked set sampling. Biom. J. 38 (5), 577-586.
Samawi, H. M. and Al-Sageer, O. A. (2001). On the estimation of the distribution
function using extreme and median ranked set sampling. Biom. J. 43(3), 357-373.
Shrahili, M., El-Saeed, A. R., Hassan, A. S., Elbatal, I., & Elgarhy, M. (2022). Estimation of
Entropy for Log-Logistic Distribution under Progressive Type II Censoring. Journal of
Nanomaterials, 2022(1), 2739606.
Sigmon, D., & Fatima, S. (2022). Fine Needle Aspiration. Treasure Island (FL): StatPearls
Publishing.
Singh, V. P. (2013). Entropy theory and its application in environmental and water
engineering. John Wiley & Sons.
Soliman, A. A., Abou-Elheggag, N. A., Abd ellah, A. H., & Modhesh, A. A. (2012).
Bayesian and non-Bayesian inferences of the Burr-XII distribution for progressive first-failure censored data. Metron, 70, 1-25.
Tahmasebi, S., Longobardi , M., Kazemi, M.R. and Alizadeh, M. (2020). Cumulative Tsallis
entropy for maximum ranked set sampling with unequal samples. Physica A: Statistical Mechanics and its Applications, 556, 124763.
Tsallis, C. (1988). Possible generalization of Boltzmann-Gibbs statistics. Journal of
statistical physics, 52, 479-487.
Wang, X., & Gui, W. (2021). Bayesian estimation of entropy for burr type xii distribution
under progressive type-ii censored data. Mathematics, 9(4), 313.
Yu, J., Gui, W., & Shan, Y. (2019). Statistical inference on the Shannon entropy of inverse
Weibull distribution under the progressive first-failure censoring. Entropy, 21(12), 1209.
Zamanzade, E. Mahdizadeh, M. (2017) Entropy Estimation From Ranked Set Samples
With Application to Test of Fit. Revista Colombiana de Estadística, 40(2), 223-241.
Zamanzade, E., Mahdizadeh, M. and Samawi, H. (2024). Nonparametric estimation of mean
residual lifetime in ranked set sampling with a concomitant variable. Journal of Applied Statistics, DOI: 10.1080/02664763.2023.2301334
Published
2025-07-01
How to Cite
Samawi, H., & Helu, A. (2025). On the inference of entropy measures under different sampling schemes. Statistics, Optimization & Information Computing. https://doi.org/10.19139/soic-2310-5070-2235
Issue
Section
Research Articles
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
