Optimization

Foundations and constrained optimization

• L-BFGS - A limited-memory quasi-Newton method for large nonlinear optimization. Approximates curvature without storing a full dense Hessian. Liu and Nocedal 1989.
• Convex optimization - Convex objectives and feasible sets make local optima global. Covers duality, optimality conditions, and numerical algorithms; convexity must be established for the actual formulation. Boyd and Vandenberghe 2004.
• Linear programming - Optimizes a linear objective under linear equality and inequality constraints. Simplex and interior-point methods address this structured problem class. Boyd and Vandenberghe 2004.
• KKT conditions - Combines feasibility, stationarity, and complementary slackness. These conditions are sufficient for convex problems; necessity requires appropriate regularity or strong duality. Boyd and Vandenberghe 2004.
• Newton and interior-point methods - Newton steps use local quadratic models. Interior-point methods solve constrained convex problems through a sequence of smooth subproblems. Boyd and Vandenberghe 2004.
• ADMM - Splits structured optimization into coupled subproblems. The review covers distributed convex learning, convergence conditions, and implementation; nonconvex extensions need separate analysis. Boyd et al. 2011.
• Frank-Wolfe / conditional gradients - Uses linear minimization over a convex feasible set instead of Euclidean projection. Jaggi gives primal-dual certificates and sparse-update analysis; the advantage depends on having a tractable linear oracle. Jaggi 2013.
• Proximal algorithms - Handles nonsmooth penalties and constraints through proximal subproblems, generalizing projection. Includes proximal gradient, acceleration, and splitting methods. Parikh and Boyd 2014.
• Mirror descent - Uses a distance induced by a convex mirror map to adapt optimization to non-Euclidean geometry. Bubeck's analysis connects the map and norm to convergence; Euclidean gradient descent is one geometry choice. Bubeck 2015.
• Coordinate descent - Approximately minimizes along individual coordinates or blocks. Reviews convergence, acceleration, and parallel variants, mainly for convex objectives. Wright 2015.

Gradient methods and adaptive updates

• Nesterov acceleration - Combines gradient steps with extrapolation. Establishes an O(1/k²) objective-gap rate for smooth convex minimization under the paper's assumptions, not arbitrary neural-network training. Nesterov 1983.
• AdaGrad - Adapts optimization geometry to observed gradients, with online regret guarantees. Useful when informative features occur at different frequencies. Duchi et al. 2011.
• AdaDelta - Adapts learning rates per dimension using first-order information. Proposed to address AdaGrad's diminishing learning rates and sensitivity to a manually chosen global rate. Zeiler 2012.
• Initialization and momentum - Studies how initialization and the momentum schedule affect training deep and recurrent networks. Poor initialization can defeat otherwise effective momentum settings. Sutskever et al. 2013.
• SAGA - Caches component gradients and uses their average to correct stochastic updates, reducing gradient variance for finite-sum objectives; supports proximal updates for nonsmooth regularization. Defazio et al. 2014.
• Adam and AdaMax - Adam combines first- and second-moment gradient estimates with bias correction; AdaMax uses an infinity-norm variant. Later convergence analyses qualify the original guarantees. Kingma and Ba 2015.
  • Nadam - Modifies Adam's momentum component using Nesterov-style acceleration. The workshop paper reports preliminary comparisons of the update with Adam. Dozat 2016.
  • AMSGrad - Exhibits a convex counterexample to Adam convergence and introduces longer-term gradient memory. Corrects weaknesses in the original analysis under specified optimization assumptions. Reddi et al. 2018.
  • AdamW - Decouples weight decay from the adaptive gradient update. L2 regularization and multiplicative weight decay are generally not equivalent in adaptive optimizers. Loshchilov and Hutter 2019.
  • TAdam - Uses a Student-t-based update to reduce sensitivity to gradient outliers. Ilboudo et al. 2020.
• RMSprop - Hinton's lecture 6e describes scaling each gradient by a running root-mean-square of recent gradients. Exponential averaging emphasizes recent magnitudes, unlike AdaGrad's cumulative sum. docs.
• Gradient descent variants - Contrasts full-batch, stochastic, and mini-batch gradients; surveys momentum, adaptive learning rates, optimization difficulties, and parallel training. Ruder 2016.
• Large-scale stochastic optimization - Connects stochastic-gradient theory to practical machine learning, including noise reduction and approximate second-order methods. Bottou et al. 2018.
• Adafactor - Factors second-moment statistics into row and column estimates for matrix parameters, reducing optimizer-state memory. Also studies update clipping and learning-rate choices. Shazeer and Stern 2018.
• Shampoo - Uses tensor-structured matrix preconditioners to capture gradient geometry beyond coordinatewise scaling. Provides regret analysis in the online convex setting. Gupta et al. 2018.
• When SGD escapes local minima - Interprets SGD through a smoothed objective. Escape and convergence results rely on assumptions about that objective and the noise; they are not general guarantees for all loss surfaces. Kleinberg et al. 2018.
• LaProp - Separates momentum from adaptive gradient normalization. Studies instability caused by coupling these components and proposes a different update order. Ziyin et al. 2020.
• SAM - Minimizes loss over a neighborhood of parameter values to seek solutions with low loss and low local sharpness. This changes the training objective and gradient computation. Foret et al. 2021.
• Lion - An optimizer discovered through program search, using momentum and sign-based updates. Its learning-rate scale differs from Adam's, so comparisons require separate tuning. Chen et al. 2023.
• D-Adaptation - Adapts step sizes without specifying an initial learning rate; distinguishes convex Lipschitz convergence guarantees from empirical SGD and Adam variants for deep learning. Defazio and Mishchenko 2023.
• Sophia - Combines momentum with an intermittently estimated diagonal Hessian and clipping. Designed for language-model pretraining; curvature estimation adds its own cost and choices. Liu et al. 2024.
• Muon for language-model training - Studies matrix-orthogonalized updates at scale, including weight decay, update scaling, and distributed implementation. Liu et al. 2025.
• Semi-gradient updates - Stop-gradient operations can produce update fields without a scalar potential. Flow-map convergence covers a case with convergent idealized dynamics.

Training schedules and online control

• Learned learning-rate controllers - Learns a policy from training statistics to adjust SGD or RMSprop rates. Transfers from small MNIST networks to tested larger networks and CIFAR-10, but the tested controller did not transfer between base optimizers. Daniel et al. 2016.
• SGDR - Cosine learning-rate decay with periodic warm restarts. Restarts raise the rate while retaining weights; cycle lengths and rate bounds are chosen in advance. Loshchilov and Hutter 2017.
• Stochastic Weight Averaging - Averages weights along constant- or cyclic-rate SGD trajectories to produce one model. The paper connects this to geometric ensembling; averaging weights differs from averaging predictions. Izmailov et al. 2018.
• Constant rate with cooldown - Shares a constant-rate training trajectory across stopping budgets through separate cooldown branches. Tuned schedules closely match cosine in the paper's language-model experiments. Hägele et al. 2024.
• Warmup and batch scaling - See Distributed Learning for the ResNet-50 recipe combining warmup and learning-rate scaling at batch size 8,192; that result does not establish a universal scaling rule.
• Schedule-Free optimization - Interpolates base and averaged iterates for gradient evaluation, replacing horizon-dependent decay. Learning rate and momentum still matter. Its AdamW variant won AlgoPerf's 2024 self-tuning track. Defazio et al. 2024; code.
  • Nonconvex Schedule-Free theory - Stationarity rates concern gradient-evaluation iterates of GD/SGD, not the averaged model used for evaluation. Assumes smooth lower-bounded losses; SGD needs unbiased bounded-variance gradients. AdamW is outside this analysis. Chae and Kim 2026.

Learning for downstream decisions

• Smart predict-then-optimize - Trains predictions for their downstream decision cost. SPO+ supplies a convex surrogate for linear-objective optimization; shortest-path and portfolio experiments show gains under model misspecification. Elmachtoub and Grigas 2022.

Bayesian optimization

• Bayesian optimization foundations - Uses a probabilistic surrogate and an acquisition function to choose expensive evaluations. Balances learning about uncertain regions against exploiting promising ones. Brochu et al. 2010.
• Practical Bayesian optimization - Studies Gaussian-process choices for hyperparameter tuning, including acquisition functions, evaluation costs, and parallel evaluations. Snoek et al. 2012.
• REMBO - Uses random embeddings to search high-dimensional objectives with low effective dimensionality. Ambient dimension alone does not determine whether this reduction is suitable. Wang et al. 2013.
  • Choosing an embedding domain - Examines the search bounds used with random embeddings. Small domains can exclude useful solutions; large domains can waste evaluations and weaken the dimensionality advantage. Binois et al. 2019.
• Bayesian optimization review - Reviews surrogate models, acquisition strategies, and extensions for automated design and machine learning. Useful for locating assumptions behind a particular BO setup. Shahriari et al. 2016.
• BOHB - Combines model-based configuration proposals with Hyperband's allocation of evaluation budgets. Intended for tuning when lower-budget runs provide useful information. Falkner et al. 2018.
• HeSBO - Uses hashing-based embeddings for Bayesian optimization in lower-dimensional subspaces. Analyzes approximation behavior and evaluates the method on high-dimensional problems. Nayebi et al. 2019.
• SIRBO - Learns an objective-relevant subspace through supervised dimension reduction, including sliced inverse regression. The embedding is informed by observed objective values. Zhang et al. 2019.
• TuRBO - Fits local probabilistic models within adaptive trust regions and allocates evaluations across regions, targeting expensive black-box optimization beyond a single global surrogate. Eriksson et al. 2019.
• SILBO - Combines evaluated points with unlabeled acquisition candidates to learn an embedding for high-dimensional Bayesian optimization; includes a randomized projection strategy. Chen et al. 2020.

Perturbation-based gradient estimation

• SPSA - Estimates a gradient from two noisy loss evaluations with simultaneous random perturbations of all coordinates. Evaluation count per estimate is independent of dimension; convergence still requires noise, smoothness and step-size assumptions. Spall 1992.
• Neural credit assignment - Perturbation estimators, random feedback, local losses and forward-mode autodiff supply learning signals with different derivative requirements and costs. See neural-network methods and comparisons.

Evolutionary and population-based search

• Differential evolution - A population-based heuristic for continuous global optimization, including nondifferentiable objectives. Storn and Price 1997.
• Evolutionary computation - Reviews search through populations, selection, and variation, including genetic algorithms and evolving programs. These methods optimize a supplied fitness criterion. Foster 2001.
• Harmony search - A heuristic inspired by musical improvisation, illustrated on traveling-salesman and engineering-design problems. Zong Woo Geem et al. 2001.
• Learning classifier systems - Reviews rule-based learning systems and their evolutionary components. Different implementations address classification, robotics, knowledge discovery, and modeling. Urbanowicz and Moore 2009.
• Memetic algorithms - Combines evolutionary search with local search components. Reviews discrete, continuous, constrained, multiobjective, and uncertain problems, emphasizing coordination between operators. Neri and Cotta 2012.
• CMA-ES - Covariance-matrix adaptation for stochastic optimization of nonlinear, nonconvex functions over continuous parameters. The tutorial derives the search strategy and its design choices. Hansen 2016.
• Population Based Training - Jointly trains a population of models and adapts their hyperparameter schedules. Combines asynchronous training with selection and exploration across population members. Jaderberg et al. 2017.

Local search and swarm methods

• Metropolis-based optimization - Approximates a constrained minimum using integral representations and Metropolis sampling. The analysis assumes a closed bounded domain and a unique minimizer. Pincus 1970.
• Simulated annealing - Connects optimization to thermal annealing: randomized moves can accept worse states, with acceptance controlled by temperature. Cooling schedule and finite evaluation budget matter. Kirkpatrick et al. 1983.
• Tabu search - Uses search memory, tabu restrictions, and aspiration criteria to guide local search. Intensification and diversification control revisiting promising regions and exploring new ones. Glover 1990.
• Particle swarm optimization - Moves candidate solutions using velocities influenced by each particle's best position and the swarm's best position. A derivative-free population heuristic for continuous search. Kennedy and Eberhart 1995.
• Artificial bee colony - A numerical optimization heuristic based on honey-bee foraging, with employed, onlooker, and scout roles balancing exploitation of food sources and exploration. Karaboga 2005.
• Cuckoo search - Combines a brood-parasitism-inspired search scheme with Lévy flights for global optimization. The cited work evaluates numerical test functions. Yang and Deb 2009.
• Firefly algorithm - Applies the heuristic to nonlinear design optimization and introduces stochastic or singular test functions with known optima. Evidence is restricted to the tested problems. Yang 2010.
• Variable neighborhood search - Systematically changes neighborhoods during descent and perturbation. Reviews basic schemes, parallel implementations, and hybrids for escaping local-search traps. Hansen et al. 2017.

Bandits and sequential allocation

• Sequential experimental design - Studies experiments whose sampling decisions depend on earlier observations. Also explains why optional stopping can invalidate tests designed for fixed sample sizes. Robbins 1952.
• Finite-time bandit analysis - Analyzes exploration-exploitation policies with logarithmic regret uniformly over time for bounded reward distributions. Guarantees concern the stated stochastic bandit setting. Auer et al. 2002.
• Empirical bandit comparison - Compares multi-armed bandit algorithms and introduces the POKER strategy. Results on the paper's reward settings are not a universal ranking of exploration policies. Vermorel and Mohri 2005.
• Bayesian probability matching - Allocates observations according to posterior probabilities that each arm is optimal. Connects Bayesian computation, adaptive experimental design, and bandit heuristics. Scott 2010.
• Bandit regret survey - Covers stochastic and adversarial payoffs, finite actions, and contextual extensions. Regret bounds depend on the feedback model and assumptions about rewards. Bubeck and Cesa-Bianchi 2012.
• Thompson sampling - Reviews posterior-sampling approaches to sequential decisions, applications with shared information across actions, and settings where the method is or is not effective. Russo et al. 2018.

Multiple objectives and evaluation

• NSGA-II - Combines nondominated sorting with elitist selection from parents and offspring, balancing fitness and spread. Approximates a set of trade-offs instead of choosing a single objective weighting. Deb et al. 2002.
• AlgoPerf - Benchmarks time to a target result across neural-network workloads on fixed hardware. Controls tuning protocols and workload variants to distinguish optimizer gains from evaluation artifacts. Dahl et al. 2023.
• LLM optimizer comparisons - In 124M FineWeb runs, longer training narrows D-Muon/SOAP gains over AdamW; AdEMAMix leads. Rankings change with batch size and tuning; the wider study reaches 720M and measures loss, not downstream abilities. Semenov et al. 2026.
• Diffusion optimizer evaluation - Across seven optimizers on CelebA-64, denoising losses span 1.8% while FID spans 6.3-15.3. Three seeds per selected configuration; tuning uses shorter runs and some grid boundaries remain unexplored. Loss rankings do not track sample quality. Bolatov et al. 2026.

Software

• SciPy optimize - Python solvers for local and global optimization, constraints, linear programming, least squares, root finding, and curve fitting. Solver choice depends on problem structure. docs.
• CVXPY - Python modeling language for expressing optimization problems and passing them to numerical solvers. Supports convex modeling and additional structured problem classes. docs.
• HiGHS - Open-source solvers for sparse linear, mixed-integer, and quadratic programming. Provides a C++ library, command-line executable, and language interfaces. docs.
• Optax - Composable gradient transformations and optimization algorithms for JAX. Supports assembling custom optimizers from smaller update components. docs.
• Sherpa - Python hyperparameter optimization for expensive iterative models, with local or cluster execution and a visualization dashboard. Hertel et al. 2020.
• pymoo - Python framework for single- and multiobjective optimization, with constraints, customizable operators, parallel evaluation, and visualization. Blank and Deb 2020.
• ensmallen - C++ numerical optimization library built around Armadillo. Includes gradient-based and derivative-free optimizers, with callbacks for monitoring and control. docs.

Related topics

• Neural Networks
• AutoML
• Meta-Learning
• Bayesian Inference
• Reinforcement Learning