Acquiring Independent Components through Hybrid PCA and ICA to Enhance the Classification Performance of Decision Tree

Abstract

The Principal Component Analysis (PCA) is widely used for modeling in both statistical and machine learning domains. However, PCA's orthogonal components may not always be independent. This research aims to compare PCA and Independent Component Analysis (ICA) using simulation and empirical data and evaluate a Decision Tree (DT) model. Two scenarios of simulation data with linear and nonlinear relationships, along with two empirical datasets were analyzed. PCA was used to project the dataset, while ICA was applied to the 6th to 10th and the 5th to 9th principal components. Both PCA and ICA resulted in projection data with zero correlation values. Scatter plots of PCA projection on nonlinear simulation data indicated consistent underlying patterns, whereas ICA projection revealed sparse patterns on both simulation datasets. The DT model utilizing 7 independent components emerged as the optimal model, displaying superior performance across accuracy, precision, recall, F1 score, Mathew's Correlation Coefficient, and Area Under Curve metrics.