AI hardware is a set of purpose-built chips—Lisp machines, GPUs, TPUs, NPUs—that accelerate AI workloads by optimising for parallelism, memory bandwidth, and low-precision arithmetic. It delivers real speed and efficiency gains, but at the cost of flexibility, portability, and architectural transparency. Its dominance reflects engineering necessity—not inevitability.
AI hardware exists because general-purpose chips were too slow and inefficient for symbolic AI—and later, neural networks.
2:58
Who runs the stack
GPUs won the training race; TPUs and NPUs carved out inference niches; by 2023, they dominate the market.
4:37
The scaling engine
Compute demand exploded—300,000× in five years—driving precision, interconnect, and memory innovation.
6:07
Where it landed
NPUs moved into laptops and phones by 2017, but remain isolated accelerators—not unified compute subsystems.
1 more chapterFewer chapters
Worth your time?
Yes. Study the whole thing.
4/ 5
What works
cutting training time for large models
enabling real-time video inference on mobile devices
reducing energy per inference by orders of magnitude
What does not
make AI models more interpretable
reduce dependency on cloud infrastructure
support arbitrary programming models out of the box
Study it if
ML engineers deploying at scale
system architects designing edge inference pipelines
chip designers building domain-specific accelerators
Skip it if
front-end developers
policy analysts assessing AI governance
researchers working primarily in symbolic logic or probabilistic programming
Who should study it, and who should skipHide who it is for
The written brief1 min read
What it is and the problem it solves
AI hardware is specialised silicon designed to execute AI programs faster and with less energy than general-purpose CPUs. It solves the bottleneck of moving data during massive parallel computation—especially for deep learning training and inference.
How it works
AI hardware uses specialised architectures—dataflow engines, systolic arrays, processing-in-memory—to accelerate matrix operations and reduce data movement. It relies on high-bandwidth memory (e.g., HBM3), optical interconnects, and low-precision arithmetic (e.g., FP8) to improve throughput and energy efficiency.
What works
GPUs have dominated since the 2010s due to parallelism and memory bandwidth. TPUs (introduced 2016) and consumer NPUs (in SoCs since 2017) deliver efficient inference. By 2019, GPUs displaced CPUs for large-scale cloud AI training. From AlexNet (2012) to AlphaZero (2017), compute increased 300,000-fold—doubling every 3.4 months.
What does not
It does not eliminate the software-hardware co-design burden. Fixed-function chips like Taalas ASICs cannot adapt to new model architectures. Processing-in-memory sacrifices memory capacity for efficiency. Event cameras and memristors alone are excluded from the definition—neither qualifies as AI hardware without full system integration.
What it changes
It shifts AI development from algorithm-first to hardware-aware engineering. It concentrates infrastructure leverage in chip vendors and cloud providers. It makes deep learning training and inference orders of magnitude faster and cheaper—but only for workloads that map cleanly to the underlying architecture.
Is it worth your time
Yes—if you train or deploy models at scale, or design systems where latency, power, or inference cost matters. For general software development or small-scale prototyping, the complexity and lock-in often outweigh gains.
