Activation Functions: From Theory to Practice

Introduction

Without activation functions, neural networks are just expensive linear algebra. They become simple matrix multiplications that can't learn anything interesting. Adding the right activation function at each layer transforms them into something powerful—able to recognize patterns in images, understand language, and beat humans at complex games.

This article explains six activation functions: what they do, how they work, and when to use each one. We'll focus on practical understanding rather than mathematical proofs.

Why Activation Functions Matter

Consider a three-layer network without activation functions. Each layer performs a linear transformation: output = W₃(W₂(W₁ × input)). This simplifies to output = W × input, where W is the product of all weight matrices. No matter how many layers you stack, the network remains linear.

Activation functions break this limitation. By introducing non-linearity at each layer, they enable the network to approximate any continuous function. This is why deep learning is possible.

Binary Step Function

The binary step function outputs 0 or 1 based on whether the input is negative or positive. This was the first activation function used in early perceptrons.

The Problem

The gradient is zero everywhere except at x = 0, where it's undefined. During backpropagation, gradients can't flow through this function, making training impossible with modern optimization algorithms. Additionally, the output provides no information about the magnitude of the input—only its sign.

Result: Obsolete. It was historically important but doesn't work with modern training methods.

Sigmoid Function

Sigmoid was the dominant activation function from the 1990s through the 2000s. It maps any input to a value between 0 and 1.

The output can be interpreted as a probability. The function is smooth and differentiable, making optimization possible with gradient descent.

The Vanishing Gradient Problem

Sigmoid works well in shallow networks, but has a critical flaw in deep networks. When x is large (say x = 5), σ(x) ≈ 0.993, and the derivative is nearly zero. During backpropagation through multiple layers, these tiny gradients multiply together. By the time the gradient reaches early layers, it's effectively zero. The network can't learn effectively.

Hyperbolic Tangent (Tanh)

Tanh is sigmoid's improved cousin, outputting values between -1 and 1 instead of 0 and 1.

Zero-Centered Output

The key improvement: tanh's output is zero-centered. When inputs to a layer are zero-centered (mean close to zero), gradients flow more efficiently. This makes optimization faster than sigmoid. However, tanh still suffers from vanishing gradients in extreme ranges, just less severely.

Rectified Linear Unit (ReLU)

ReLU is deceptively simple: it outputs the input if positive, zero if negative.

Despite its simplicity, ReLU revolutionized deep learning. In 2012, a paper about deep convolutional neural networks using ReLU showed that networks could be trained 10x deeper, 100x faster than with sigmoid. This was the breakthrough moment for modern deep learning.

Why ReLU Works

For positive inputs, the gradient is always 1. This means backpropagation works efficiently through deep networks. The vanishing gradient problem? Solved. For negative inputs, the neuron is inactive (output zero), which creates sparsity—only some neurons activate for a given input. This can improve computational efficiency and generalization.

The Dead Neuron Problem

If a neuron receives mostly negative inputs during training, its weights may never update (since gradient is zero for negative inputs). The neuron becomes "dead"—permanently inactive. This led to variants like Leaky ReLU (f(x) = 0.01x for x < 0) that allow small gradients even for negative inputs.

Linear Activation

Linear activation (also called identity function) simply outputs the input: f(x) = x. It's exclusively used in the output layer for regression tasks.

When to Use

For regression problems where output can be any real number (predicting house prices, stock prices, temperature), a linear output layer is essential. The network's hidden layers use non-linear activations (ReLU, tanh), but the output layer must be linear to allow unbounded predictions.

Using sigmoid or tanh in the output layer would incorrectly constrain predictions to a fixed range.

Softmax Function

Softmax converts a vector of arbitrary values into a probability distribution where all outputs sum to 1.

Unlike sigmoid (which handles binary classification), softmax handles multiple classes. For a 10-class classification problem, the output layer has 10 neurons. Softmax converts their raw outputs into probabilities summing to 1.

The exponential in the numerator emphasizes differences between values. If one value is much larger, softmax assigns it most of the probability mass. This "winner-take-all" property makes softmax ideal for classification.

Practical Recommendations

For most problems: Use ReLU in hidden layers. Choose the output activation based on the task—sigmoid for binary classification, softmax for multi-class, and linear for regression.