Benchmarking Metaheuristics for Neural Network Optimization: A Comprehensive Study

Abstract

Deep learning is a machine learning method that has been successfully applied in various applications. However, these methods have structures and parameters that vary according to the specific application. Training a neural network is a highly complex task, but it is crucial to the process of learning the network. This step is generally carried out using gradient descent methods. Alternatively, metaheuristics are approximate optimization algorithms grounded in specific theories and objective functions that have shown better results in various domains. The idea of this work is therefore to use metaheuristics to train a deep neural network and compare the obtained results. To evaluate the different metaheuristics, we tested them on six different machine learning datasets. We have found that the PSO algorithm gives the best results in the training data. While its accuracy deteriorates on the testing data, we can conclude that the PSO is prone to overfitting. On the other hand, simulated annealing, and genetic algorithms generalize better to the test data. Comparing the execution times of the algorithms, we have found that simulated annealing is the fastest. This makes it the algorithm with the best trade-off between execution time and accuracy.