GPUs, TPUs, and NPUs all exist to do the same job, multiplying large grids of numbers very fast, but they differ in who can use them, how code reaches them, and what happens when a model does something unusual. This explainer answers a practical question: when a company says a feature runs on "the NPU" or a model trained on "TPUs," what does that actually imply compared with a GPU?
This explainer is based on the chip makers' own documentation and research pages, verified as of 13 October 2026. It builds on our earlier piece on on-device AI and on the cost pressure covered in AI bubble indicators.
The three chips, side by side
| GPU (NVIDIA Hopper example) | TPU (Google Cloud) | NPU (Apple, Qualcomm) | |
|---|---|---|---|
| Core idea | Many programmable cores, plus Tensor Cores for matrix multiply-accumulate | Custom ASIC built around a 128x128 systolic array (the MXU) | Low-power block for neural-network layers, beside a CPU and GPU |
| Where you meet it | Data centers, workstations, PCs | Google Cloud | Inside phones, tablets, laptops |
| How code reaches it | CUDA; custom operations are normal | XLA compiler turns the linear-algebra part of a graph into TPU code | Vendor frameworks; Apple's reference needs specific tensor layouts |
| Stated sweet spot | General parallel compute plus AI | Long training runs, large batches, matrix-dominated models | "Sustained" low-power inference |
| Stated weak spot | Not stated in the sources read | Dynamic shapes, custom ops, heavy branching | Bandwidth-bound at short sequences; layout copies |
What they share: one operation
NVIDIA describes its Hopper Tensor Cores as "specialized high-performance compute cores for matrix multiply and accumulate (MMA) math operations." Google's TPU documentation centers on the matrix unit, a 128x128 systolic array that the XLA compiler tiles matrix multiplies to fit. Qualcomm describes neural-network layers as a mix of scalar, vector, and tensor math, and says its Hexagon NPU is built "for sustained, high-performance AI inference at low power." Three different phrasings, one underlying reason: transformer models spend most of their compute on matrix multiplication, so each design puts dedicated silicon there.
Where they diverge: who writes the code
The sharpest difference is not speed. It is how much freedom the programmer keeps.
- GPU: NVIDIA's Hopper material emphasizes programmability. Thread block clusters and distributed shared memory are extensions to the CUDA model, so a developer can bring unusual operations and still run on the same chip.
- TPU: Google's documentation says code must be compiled with XLA, and lists workloads to avoid: models with custom operations in the main training loop, dynamic tensor shapes (recompilation is slow), frequent branching, and high-precision arithmetic. For those, the same page points to CPUs or GPUs.
- NPU: Apple's research page on running transformers on its Neural Engine shows the constraint in detail. The engine favors 4D, channels-first tensors, the last axis must be contiguous and 64-byte aligned, and a singleton last axis can waste up to 32x memory at 16-bit. Reshapes and transposes can trigger copies.
The pattern: the more specialized the chip, the more the model has to be shaped to fit it. That is the trade for efficiency.
Why the headline numbers cannot be lined up
Chip announcements quote throughput in different units, and the unit hides the precision.
| Source | Figure as stated | What is missing for comparison |
|---|---|---|
| Apple Neural Engine research page | 15.8 TFLOPS (FP16) for the fifth-generation 16-core engine | The page is an older article; the engine is not the current-generation chip |
| Qualcomm Hexagon product page | "45 trillion operations per second" | The text read does not state the numeric precision or workload |
| NVIDIA Hopper article | FP8 halves storage and doubles throughput versus FP16 or BF16 | Preliminary 2022 specifications for an earlier GPU generation |
Compare 15.8 FP16 TFLOPS with 45 "operations per second" and you are comparing a precision-specific number with an unspecified one. NVIDIA's own statement shows why that matters: dropping from FP16 to FP8 doubles throughput on the same silicon. A bigger number at lower precision is not a faster chip. When a spec sheet omits the precision, treat the figure as a marketing ceiling, not a benchmark.
Capacity and memory: the other bottleneck
NVIDIA lists 80 GB of HBM3 with over 3 TB/sec of bandwidth on the SXM5 Hopper GPU. Google's documentation says TPU on-chip high-bandwidth memory supports larger models and batch sizes. Apple, by contrast, reports that many NPU configurations become bandwidth-bound at short sequence lengths and suggests larger batches or quantization and pruning. In plain terms, data center chips are limited by how big a model they can hold; phone chips are limited by how fast they can feed the compute block.
Training versus inference: the split that sorts the chips
Google's guidance for TPUs leans toward training: long-running jobs measured in weeks or months, large models, and large effective batch sizes. Apple's Neural Engine page and Qualcomm's NPU article both describe inference, running a model that already exists. A phone never needs to train a large model; it needs to run one repeatedly without draining the battery or overheating. That difference in job explains most of the design choices above: the data center chip is built to stay saturated for weeks on a huge batch, while the device chip is built to finish one small request quickly and go back to idle. It also explains why a vendor can truthfully call a chip fast and still not mean the same thing as a competitor who used the same word.
What each one is not for
- TPU: not a drop-in for arbitrary code. Google's page also notes that the Cloud TPU API is deprecated, directing users to Compute Engine, Google Kubernetes Engine, or the Gemini Enterprise Agent Platform. Current generations listed include TPU7x (Ironwood), v6e, v5p, v5e, and v4.
- NPU: not for training large models. The sources describe inference. Qualcomm frames it as one part of heterogeneous computing, where the CPU handles sequential control and the GPU handles streaming parallel data.
- GPU: not what Qualcomm points to for battery life: it argues that splitting work across CPU, GPU, and NPU improves thermal efficiency and battery life, though it gives no measurements.
Which one matters for you
- You train or serve very large models and use Google Cloud: TPUs are documented as a Cloud service, with strong fit for large-batch, matrix-heavy training and caveats for dynamic or custom-op models.
- You need flexibility, custom operations, or your own hardware: GPUs, because of the programming model.
- You are choosing a phone or laptop for AI features: look for an NPU, but compare features supported and software maturity rather than a single TOPS figure.
- You are reading a spec sheet: find the precision behind every throughput number before comparing.
Frequently asked questions
Is an NPU just a small GPU?
No. Qualcomm describes the NPU as built from the ground up for low-power AI inference, while the GPU handles streaming parallel data in the same chip. They are separate blocks with different jobs.
Can I buy a TPU?
Google's documentation presents TPUs as a Google Cloud offering, accessed through services such as Compute Engine and Google Kubernetes Engine, and the pages read do not describe a retail purchase.
Why do TPUs struggle with dynamic shapes?
Because the XLA compiler builds code for specific tensor shapes. Google's documentation says dynamic shapes force slow recompilation.
Is a higher TOPS number always better?
No. NVIDIA's own article shows FP8 doubling throughput versus FP16 on the same hardware, so a figure is only meaningful alongside its numeric precision.
Last verified: 13 October 2026. Chip generations and specifications change often; several sources cited here describe earlier hardware generations, as noted above.