technologybriefs
10:27in productionCh. 1 · Not invented for AI/ 10:27 · ceiling 15 min
AI

Backpropagation

1970

Backpropagation is not learning—it’s bookkeeping for learning.

Backpropagation is a precise, narrow, and mechanically elegant solution to one computational bottleneck: gradient calculation in feedforward networks. It is neither AI nor learning—but the indispensable arithmetic that made both possible.

AIAlgorithmsArtificial intelligenceComputingAlgorithms across the network →
Chapters & takeaways6
  1. 1:06
    Not invented for AI

    The 1970 publication was not about neural networks—it was reverse-mode automatic differentiation for nested differentiable functions.

  2. 2:12
    Reverse accumulation, not intuition

    It saves time by reusing intermediate derivatives—computing one backward pass instead of many forward ones.

  3. 3:19
    Gradient calculator, not learning algorithm

    It computes gradients only. Everything else—optimisation, architecture, evaluation—is separate and external.

  4. 4:35
    Two rediscoveries, one tool

    Werbos and Rumelhart independently adapted it to MLPs in 1982—decades after its mathematical formulation.

  5. 5:50
    Tangled origins

    Its history is fragmented: multiple independent derivations, inconsistent naming, and blurred boundaries with adjacent methods.

  6. 7:00
    Narrow scope, precise output

    It only works for feedforward networks with differentiable components and a scalar loss.

Worth your time?

Yes. Study the whole thing.

4.5/ 5
What works
  • computes exact gradients efficiently
  • avoids redundant derivative calculations
  • enables iterative weight updates in multilayer networks
What does not
  • generalise beyond differentiable feedforward networks
  • specify how gradients are applied
  • guarantee convergence
  • address data quality or quantity
Study it if
  • researchers implementing custom neural network training loops
  • engineers debugging gradient computation
  • historians of automatic differentiation
Skip it if
  • product managers evaluating AI readiness
  • policy makers assessing AI risk
  • students seeking intuitive models of learning
The written brief1 min read

What it is and the problem it solves

Backpropagation is an efficient gradient-computation method for feedforward neural networks. It solves the problem of calculating weight-space gradients of a loss function quickly enough to make iterative parameter updates feasible.

How it works

It applies the chain rule in reverse, layer by layer, from output to input, computing gradients of the loss with respect to each weight without redundant calculations.

What works

It correctly computes the exact gradient of the loss with respect to all weights for a single example, given differentiable layers and a defined loss. Its reverse accumulation avoids exponential recomputation inherent in naive chain-rule application.

What does not

It does not specify how gradients are used—no optimiser, no learning rate, no regularisation. It does not generalise to arbitrary computation graphs without modification. It does not solve vanishing gradients, overfitting, or data scarcity.

What it changes

It enables practical training of multilayer perceptrons by making gradient computation tractable for deep networks—shifting neural network research from theoretical curiosity to empirical engineering.

Is it worth your time

Yes—if you train feedforward neural networks and need efficient gradient computation. It is not a learning algorithm itself, and offers no convergence guarantees or architectural insight.

Same field · AI4 of 18
10:06
Convolutional neural network1988CNNs solve image recognition by learning local, shared filters—cutting parameters, stabilising training, and replacing hand-crafted features. They work reliably on digit classification and scaled contest tasks—but demand data, compute, and architectural discipline. Not a universal solution. A specific, effective scaffold.
9:18
Diffusion modelDiffusion models are a 2015 method for sampling from complex distributions using thermodynamic-inspired noise reversal. They work by learning to undo a diffusion process—turning noise into data, step by step. They succeed in computer vision tasks as of 2024. They do not generalise beyond vision. They require iterative sampling, trading speed for quality. Their mechanism is physical in inspiration, statistical in execution, and narrow in current utility.
9:46
Expert systemExpert systems encode human expertise as if–then rules. They work only where domain logic is stable, complete, and expressible in discrete conditions. Their success depends entirely on expert authorship — not algorithmic power. They are obsolete as general AI tools, but remain relevant as deterministic decision engines in tightly bounded, high-stakes domains.
9:11
Generative adversarial network2014GANs are an adversarial framework for unsupervised generation. They work by pitting two neural networks against each other: a generator learns to produce data that fools a discriminator, which itself improves at spotting fakes. This indirect training yields samples statistically aligned with the training set—including photorealistic images that pass superficial human inspection. But GANs offer no guarantees on semantic coherence, stability, or interpretability. Their value lies in enabling distribution learning without labels—not in reliability, control, or fidelity.
Up next in Technology

Battery management system

· 10:17

A BMS doesn’t make batteries smarter — it makes them less likely to catch fire while pretending to be intelligent.

Play nextBack to AI
10:17