Activation Functions | AI Learning Guide

Once a neuron has processed its inputs, it needs to decide what to do next. This is the job of the 'activation function'. It acts like a gatekeeper. If the signal is strong enough, it lets it pass; if not, it blocks it. Crucially, these functions are non-linear. This means the network can learn complex, curved relationships in data, rather than just simple straight lines. Common examples include ReLU, which is fast and efficient, and Sigmoid, which is often used for probabilities.

0:00 / 0:00

Auto-play

The Decision Gate

After a neuron calculates the weighted sum of its inputs plus the bias, it passes this number through an 'activation function'. This function decides whether the neuron should 'fire' or not. Without activation functions, a neural network would just be a series of simple linear calculations, unable to solve complex problems like recognizing handwritten digits or understanding speech.

Why Non-Linearity Matters

Real-world data is rarely a straight line. Activation functions introduce 'non-linearity', allowing the network to draw curved boundaries between different categories. For example, distinguishing between cats and dogs requires recognizing complex, non-linear shapes in images.

Common Activation Functions

Different functions serve different purposes:

ReLU (Rectified Linear Unit): The most common choice; it outputs the input directly if positive, otherwise zero.
Sigmoid: Squashes numbers between 0 and 1, useful for probability outputs.
Softmax: Used in the output layer for multi-class classification to ensure probabilities sum to 1.