Optimization

Optimization Techniques: Training Networks Faster

Updated February 2026 12 min read

Introduction

Neural networks learn through gradient descent: compute gradients, update weights, repeat. But simple gradient descent is slow. For large networks training on massive datasets, slow means impractical.

Optimization techniques accelerate training by making gradient descent smarter. Some preprocess data, others adjust how gradients are applied, and some use adaptive learning rates. Together, they can reduce training time from weeks to hours while improving final performance.

Feature Scaling

When input features have vastly different scales (height in meters vs. age in years), gradient descent struggles. The loss function becomes elongated, forcing the optimizer to take smaller steps.

Standardization

Transform each feature to have mean 0 and standard deviation 1:

x_normalized = (x - mean) / std

This puts all features on the same scale, allowing the optimizer to find the direction of steepest descent more directly.

Normalization

Scale features to a range, typically [0, 1]:

x_normalized = (x - min) / (max - min)

Advantages

Faster convergence
Better gradient flow
Simple to implement

Disadvantages

Requires knowing data statistics
Must be applied consistently

Batch Normalization

Feature scaling normalizes inputs. Batch normalization normalizes the outputs of every layer during training.

y_normalized = (y - batch_mean) / sqrt(batch_var + epsilon)

By normalizing layer inputs, we keep them in favorable ranges for activation functions, reducing the internal covariate shift problem. This allows higher learning rates and faster training.

Advantages

Faster training
Reduces sensitivity to weight initialization
Acts as regularization
Allows higher learning rates

Disadvantages

Adds computational cost
Behavior differs during training vs. testing
Requires tracking running statistics

Mini-Batch Gradient Descent

Standard gradient descent (SGD) computes gradients on one sample, which is noisy. Full-batch gradient descent computes gradients on all samples, which is stable but slow.

Mini-batch gradient descent splits the dataset into small batches (typically 32-256 samples) and updates weights based on each batch. This balances noise and stability.

                Batch size is a critical hyperparameter. Larger batches are more stable but require more memory. Smaller batches are noisier but allow more frequent updates.
            

Momentum

Standard gradient descent updates weights with the current gradient. Momentum adds history: use a weighted average of past gradients.

velocity = beta * velocity + gradient
weights = weights - learning_rate * velocity

Think of it as rolling a ball downhill. The ball gains momentum and accelerates in consistent directions while moving slowly in oscillating directions. This helps escape local minima and converges faster.

                β (typically 0.9) controls how much history to remember. Higher β means more momentum.
            

RMSProp

RMSProp adapts the learning rate per parameter based on the magnitude of recent gradients. Parameters with large gradients get smaller updates (lower effective learning rate).

rms = beta * rms + (1 - beta) * gradient²
weights = weights - learning_rate * gradient / sqrt(rms + epsilon)

This helps with parameters that consistently have large or small gradients, allowing the optimizer to use appropriate step sizes per parameter.

Advantages

Adaptive learning rates
Works well in practice
Handles different gradient magnitudes

Disadvantages

More hyperparameters to tune
More memory for storing running averages

Adam Optimizer

Adam combines momentum and RMSProp. It maintains both a running average of gradients (momentum) and a running average of gradient magnitudes (adaptive learning rates).

m = beta1 * m + (1 - beta1) * gradient
v = beta2 * v + (1 - beta2) * gradient²
weights = weights - learning_rate * m / sqrt(v + epsilon)

Adam is the most popular optimizer in modern deep learning. It works well across different architectures and datasets with minimal tuning.

Advantages

Fast convergence
Robust to different problems
Industry standard
Minimal hyperparameter tuning

Disadvantages

Can sometimes overfit
Requires more memory
Sometimes performs worse than SGD + momentum on some tasks

Default hyperparameters work well: learning_rate=0.001, beta1=0.9, beta2=0.999.

3D surface unavailable — WebGL could not start. The loss curves alongside remain accurate.

Learning rate

Drag the surface to orbit; toggle any optimizer with the chips above. The surface is illustrative — a synthetic function chosen to contain a plateau, two local minima and one global minimum, because real loss landscapes have millions of dimensions and cannot be drawn. The optimizers are not. Each runs its real update rule on the analytic gradient of that surface, so every difference you see is produced by the algorithm itself. Try the Plateau start to watch SGD stall where Adam does not — then try Shallow, where the adaptive methods settle for a worse minimum than plain SGD.

Learning Rate Decay

A fixed learning rate can be too aggressive early in training and too conservative late in training. Learning rate decay reduces the learning rate over time.

learning_rate = initial_lr * decay_rate^epoch

With decay, the optimizer takes larger steps initially to explore the loss landscape, then smaller steps to fine-tune weights. This often results in better final solutions.

Common strategies include exponential decay (reduce by a fixed percentage each epoch), step decay (reduce by a factor every N epochs), and cosine decay (follow a cosine curve).

Practical Pipeline

Stage	Technique	Impact
Data Preparation	Feature Scaling	2-3x faster training
Training Setup	Mini-Batch + Batch Norm	Stable, fast convergence
Optimization	Adam Optimizer	Usually converges faster
Fine-tuning	Learning Rate Decay	Better final performance

Recommended Training Setup

Start with: standardized features + mini-batch (32-256) + Adam optimizer (lr=0.001) + batch normalization.

If training oscillates, add momentum or try learning rate decay. If training is too slow, increase batch size. If underfitting, increase learning rate. If overfitting, apply regularization (covered in another article).

Conclusion

Modern optimization isn't just about gradient descent—it's about preprocessing data, using adaptive learning rates, normalizing layer activations, and decaying learning rates over time.

The difference between a network that takes weeks to train and one that trains in hours comes down to these optimization techniques. Adam combined with batch normalization is the industry standard for good reason: it works across diverse problems with minimal tuning.

But these aren't one-size-fits-all. Different problems may require different approaches. The key is understanding what each technique does and when to apply it.