technologybriefs
9:29in productionCh. 1 · What problem does RLHF solve?/ 9:29 · ceiling 15 min
Models · AI

Reinforcement learning from human feedback

RLHF outsources ethics to annotators—and then optimises for their rankings, not their intent.

RLHF replaces hand-coded rewards with learned ones—using human rankings as ground truth. It enables alignment at scale for language models, but only within the narrow bounds of the annotation task. It is widely used, not universally reliable.

ModelsAIAlgorithmsArtificial intelligenceMachine learningAlgorithms across the network →
Chapters & takeaways4
  1. 0:56
    What problem does RLHF solve?

    RLHF is not about teaching machines right from wrong—it's about building a reward proxy from limited human feedback.

  2. 2:14
    How does RLHF actually work?

    Human rankings become a supervised signal for a reward model, which then drives policy updates via PPO.

  3. 3:52
    Where did RLHF succeed?

    It gained traction through OpenAI’s InstructGPT paper—not because it was novel, but because it scaled alignment to real-world prompting.

  4. 5:38
    What does RLHF assume—and what breaks when it’s wrong?

    The reward model is trained directly on feedback, not inferred indirectly—making it sensitive to annotation noise and framing effects.

Worth your time?

Yes. Study the whole thing.

3.5/ 5
What works
  • text continuation
  • summarisation
  • instruction following (InstructGPT)
What does not
  • eliminate reward hacking
  • guarantee generalisation beyond ranked prompt distribution
  • ensure annotation consistency across domains or time
Study it if
  • teams deploying instruction-tuned LMs
  • researchers studying preference learning
  • practitioners needing rapid alignment iteration
Skip it if
  • systems requiring certified safety guarantees
  • low-resource annotation environments
  • real-time, high-stakes decision-making
The written brief1 min read

What it is and the problem it solves

RLHF is a technique to align intelligent agents with human preferences. It solves the problem of specifying complex, context-sensitive goals when explicit reward engineering fails.

How it works

RLHF trains a reward model on human rankings of responses, then uses that model as a reward function to update a policy via proximal policy optimization.

What works

It works where human judgements are ordinal, repeatable, and cover representative prompt-response pairs—demonstrated in text continuation and instruction-following tasks like InstructGPT.

What does not

It does not eliminate reward hacking, nor does it guarantee generalisation beyond the distribution of ranked prompts; it assumes human rankings are stable, consistent, and sufficient to capture preference structure.

What it changes

It shifts alignment from hand-crafted reward functions to learned proxies, making preference-based tuning feasible for large language models—but at the cost of opacity, annotation dependency, and reward-model brittleness.

Is it worth your time

Yes—if you are aligning language models with nuanced preferences and have access to consistent human annotators, but not if you need precise, auditable, or scalable alignment without annotation overhead.

Same field · Models4 of 5
9:18
Diffusion modelDiffusion models are a 2015 method for sampling from complex distributions using thermodynamic-inspired noise reversal. They work by learning to undo a diffusion process—turning noise into data, step by step. They succeed in computer vision tasks as of 2024. They do not generalise beyond vision. They require iterative sampling, trading speed for quality. Their mechanism is physical in inspiration, statistical in execution, and narrow in current utility.
8:14
Gaussian splattingGaussian splatting is a volume rendering technique introduced by Lee Westover in the early 1990s and revitalised in 2023 by Inria for real-time radiance field rendering. It uses differentiable, anisotropic, explicit 3D Gaussians integrated from sparse camera-calibrated points. It is rasterisation-based, GPU-optimised, and avoids surface or line primitives. It enables fast photorealistic reconstruction from few images—but offers no mesh output, no topological guarantees, and limited editability.
9:11
Generative adversarial network2014GANs are an adversarial framework for unsupervised generation. They work by pitting two neural networks against each other: a generator learns to produce data that fools a discriminator, which itself improves at spotting fakes. This indirect training yields samples statistically aligned with the training set—including photorealistic images that pass superficial human inspection. But GANs offer no guarantees on semantic coherence, stability, or interpretability. Their value lies in enabling distribution learning without labels—not in reliability, control, or fidelity.
9:28
LoRA (machine learning)LoRA makes fine-tuning large language models radically cheaper to train without slowing inference—by replacing full-weight updates with low-rank matrix additions. It works. It scales. It merges. But it assumes weight updates are low-rank—and that assumption fails where tasks demand dense, global parameter shifts.
Up next in Technology

Relativity Space

2015 · 10:36

A rocket company betting $1.335 billion on printing more before flying any.

Play nextBack to Models
10:36