Improved Gradient Descent Optimization Using Adaptive Step-Sizing

Abstract

Gradient Descent (GD) is among the classic and most common optimization algorithms in machine learning, signal processing and numerical optimization. Although conceptually appealing and simple, traditional gradient descent with a constant learning rate is prone to slow convergence, oscillation or divergence, particularly in high-dimensional non-convex or ill-conditioned optimization problems. The choice of a suitable size of steps (learning rate) is also a critical issue since the progressively small step sizes results in very sluggish convergence, and conversely, the large step size can overshoot   the minima. This paper describes a creation of a better gradient descent based on adaptive step-sizing, which varies the learning rate at each training step based on the gradient tendencies and previous update data. The suggested methodology will be designed to make convergence faster, achieve better numerical stability, and have more robustness in a broad longitude of optimization problems. We present a complete formulation of theory, the design of algorithms and their comparison with classical gradient descent and other popular adaptive versions. The presence of extensive experiments on benchmarks of optimization functions and machine learning tasks demonstrate that adaptive step-sizing has a substantial positive effect both on the rate of convergence and the quality of the solution and on the sensitivity to hyper parameter++ starting point. The findings prove that when used in contemporary optimization problems with large scale, adaptive gradient descent techniques are feasible and scaled to large scale problems. The contribution of this work is a framework that is well structured and can be easily incorporated into the existing learning systems and optimization pipelines.