Edge AI & Robot Compute: The Ultimate Guide

Every robot carries a small data center it has to power, cool, and keep from missing deadlines. A humanoid walking across a warehouse floor is running a 1 kHz balance loop on one processor, a 50 Hz learned locomotion policy on another, a stereo-depth pipeline chewing through 60 frames a second on a third, and maybe a vision-language model deciding what to do next on a fourth, all inside a chassis with a fixed battery, a fixed thermal envelope, and no fan noise budget in a room full of people. Get the compute architecture wrong and the robot either falls over because the control loop jittered, or thermal-throttles into a slideshow ten minutes into a shift, or drains its pack in twenty minutes because someone put a 60 W GPU where a 5 W accelerator would have done.

This guide is for the people who build that stack: the roboticist deciding whether a task needs a microcontroller, a system-on-chip, or a GPU module; the ML engineer whose model runs fine on a workstation and falls apart on the robot; and the systems person making hard-real-time control and best-effort AI inference coexist on one machine without one starving the other. We cover the tiers of onboard compute and what each is for, how to split deterministic control from stochastic inference, how to size a processor against a workload, the model-optimization toolkit that fits a training-time network into the power and latency budget, and the cloud-edge division of labor.

The take: Onboard robot compute is a heterogeneous system sized by three hard budgets that all bind at once: latency (the control loop cannot miss its deadline), power (the battery is fixed), and thermal (the heat has to go somewhere). You partition the workload across tiers: an MCU for the hard-real-time loop, an SoC or GPU/NPU for perception and learned policies, sometimes an FPGA for fixed-function sensor work, and you keep the deterministic part physically and temporally isolated from the best-effort AI part. The model-optimization work (quantization, pruning, distillation, compilation) exists to move an AI workload from "fits on a workstation" into "fits in the robot's remaining watts and milliseconds." Cloud offload is for the slow, heavy, non-safety-critical layer only. Anything a person or a wall could get hurt by stays local.

Companion reading: real-time control systems, ROS 2, robot sensors, machine vision, and foundation models & VLAs for robotics.

Table of contents

1. Key takeaways
2. Compute as a system-design problem
3. The compute tiers: MCU, SoC/SBC, GPU/NPU, FPGA
4. Splitting hard-real-time control from AI inference
5. Sizing compute: the roofline budget
6. Model optimization: quantization, pruning, distillation, compilation
7. Power and thermal budgets
8. Sensor bandwidth and the latency chain
9. ROS 2 on the edge
10. The cloud-edge division of labor
11. Representative hardware categories
12. Failure modes and outlook
13. Frequently asked questions

Compute as a system-design problem

The instinct from the datacenter world is to pick the processor with the most throughput and move on. On a robot that instinct gets you a machine that cannot ship. Onboard compute is constrained by three budgets that all have to close at the same time, and they trade against each other.

Latency. A robot is a real-time system. Some loops have hard deadlines: the balance controller on a biped, the current loop on a motor, the emergency-stop logic. Miss the deadline and the physical system misbehaves, sometimes catastrophically. The latency that binds here is worst-case latency (the tail), because the one time in ten thousand that the loop runs long is the time the robot falls. Average latency barely enters into it. This is a different discipline from throughput optimization, and it is why the fast, throughput-optimized accelerators that dominate AI benchmarks are exactly the wrong thing to put a hard-real-time loop on. See real-time control systems.

Power. A mobile robot runs off a fixed battery. Every watt the computer draws is a watt not available for locomotion and a direct hit to runtime. A 60 W compute module on a robot with a 500 Wh pack is over 10% of the energy budget before a single motor turns. Compute power also does not scale down gracefully: an idle GPU still draws meaningful static power, so a lightly used oversized accelerator wastes energy continuously.

Thermal. Every watt of compute becomes a watt of heat that has to leave the chassis. Robots are often sealed for ingress protection, run in warm environments, and cannot always tolerate fans (noise near people, dust ingestion, moving parts to fail). A processor rated at 40 W is only usable at 40 W if you can actually reject 40 W of heat continuously, and on a sealed fanless enclosure you frequently cannot. Silicon protects itself by throttling: it drops clocks when it gets hot, so a chip that benchmarks fast for thirty seconds delivers a fraction of that sustained.

Rule of thumb: Size compute against the sustained thermal design point in the robot's real enclosure and ambient, not against the peak benchmark number on a bench with a fan. The gap between the two is routinely 2x or more.

These three braid together. Lowering power lowers heat and extends runtime but usually costs latency headroom. Adding a heatsink buys sustained performance but adds mass, which costs runtime and payload. The art of robot compute is finding the partition of the workload across processors that closes all three budgets at once. That partition is the subject of the rest of this guide.

The compute tiers: MCU, SoC/SBC, GPU/NPU, FPGA

Robot compute is a stack of tiers, each good at something the others are bad at. A real robot uses several.

Microcontroller (MCU). A small, deterministic processor (an ARM Cortex-M or R class part, an ESP32, a motor-control DSP) running bare-metal or an RTOS. Clock speeds of tens to hundreds of MHz, memory in kilobytes to low megabytes, power in the milliwatt-to-single-watt range. It has no operating system to introduce jitter, so its worst-case timing is tight and predictable. This is where hard-real-time control lives: current loops, PID, safety interlocks, sensor sampling at precise intervals. An MCU cannot run a neural network of any size, and that is fine, because that is not its job. See the real-time control loop discussion in real-time control systems.

System-on-chip / single-board computer (SoC/SBC). A general-purpose applications processor running Linux: multi-core ARM (or x86) at 1-3 GHz, gigabytes of RAM, often with an integrated GPU and increasingly an NPU. This is the robot's main brain for everything that is not hard-real-time: sensor fusion, SLAM, planning, the ROS 2 graph, coordination. It is fast and flexible but non-deterministic (Linux scheduling, memory management, and caches introduce jitter), so it is the wrong place for a hard deadline unless you carve out an isolated real-time core.

GPU / NPU. Throughput accelerators for the AI workload. A GPU (the discrete or integrated parallel processor) or an NPU (a neural processing unit, a fixed-function matrix-multiply engine) exists to run neural networks fast: convolution, attention, the dense linear algebra of perception and learned policies. They deliver enormous throughput (trillions of operations per second) at the cost of high power and poor worst-case latency. On robots these usually appear as an integrated block on the SoC (a Jetson-class module, an Apple-silicon-class SoC, a Qualcomm robotics platform) rather than a separate card, because power and space are tight.

FPGA. A field-programmable gate array: reconfigurable logic you wire into a custom circuit. FPGAs shine at fixed-function, high-bandwidth, low-latency stream processing: stereo-depth computation, image signal processing, sensor pre-processing, motor control with microsecond determinism. They give you hardware-level parallelism and predictable timing without a CPU in the loop. The cost is development effort (you are describing hardware, not writing software) and they are less flexible once deployed. They appear most in high-volume products, vision pipelines, and aerospace/defense where determinism is non-negotiable.

Tier	Clock / throughput	Power	Determinism	Runs NNs?	Typical job
MCU	10s-100s MHz	mW-1 W	Very high (bare-metal/RTOS)	Tiny only	Current loop, PID, safety, sensor timing
SoC / SBC	1-3 GHz, multi-core	5-30 W	Low (Linux)	Small on CPU	Perception, planning, ROS 2, fusion
GPU / NPU	10s-100s TOPS	5-60 W	Poor (throughput-optimized)	Yes, the main engine	Vision, learned policy, VLA inference
FPGA	Fabric-dependent	2-20 W	Very high	Fixed-function accelerators	Stereo depth, ISP, deterministic motor control

Rule of thumb: Map each task to the cheapest tier that meets its determinism and throughput needs. A 1 kHz control loop belongs on an MCU or real-time core, not a Linux thread. A 30 Hz object detector belongs on an NPU, not a CPU. Putting a task on a higher tier than it needs wastes power; putting it on a lower tier than it needs misses deadlines.

Splitting hard-real-time control from AI inference

This is the central architectural decision, and it follows directly from the tiers. Hard-real-time control and AI inference have opposite requirements, and forcing them onto the same processor makes both worse.

Control wants determinism: a guaranteed, bounded response every single cycle, even if the average is unremarkable. AI inference wants throughput: the most work per second on average, and it achieves that with deep pipelines, large caches, dynamic memory, and batching, all of which make worst-case latency terrible and unpredictable. A garbage-collection pause, a page fault, a cache eviction, or a big convolution monopolizing memory bandwidth is a non-event for a vision model and a fall for a balancing robot.

The standard answer is to physically or logically separate the two.

Two-processor split (most common). An MCU (or a real-time coprocessor) runs the hard-real-time control loop entirely on its own, sampling sensors and commanding actuators at a fixed rate (say 1 kHz) with tight jitter. A separate Linux SoC runs perception and the learned policy at a slower rate (10-100 Hz). The two talk over a deterministic link, EtherCAT, CAN/CAN-FD, SPI, or a shared-memory channel, exchanging setpoints and state. The AI side can stutter, throttle, or crash without the control side missing a beat, because the control side is a separate computer. Many quadrupeds and humanoids are built exactly this way.

Asymmetric multiprocessing on one SoC. Modern SoCs pair application cores (Cortex-A running Linux) with a real-time core (Cortex-R or Cortex-M) on the same die. You pin the hard-real-time loop to the R-core running an RTOS or bare-metal, and Linux with the AI stack runs on the A-cores, communicating over shared memory. One chip, two worlds, isolated by hardware.

Real-time Linux with core isolation. If you must run everything on one Linux SoC, use the PREEMPT_RT patch, isolate CPU cores for the control thread (isolcpus), pin the thread with real-time priority (SCHED_FIFO), lock its memory (mlockall to prevent paging), and keep the AI workload off those cores and off the memory bandwidth they need. This is workable up to moderate rates but is the hardest to get right, because the AI workload and the control loop still contend for the shared memory controller and last-level cache. A heavy convolution can starve the control thread of bandwidth even when it has its own core.

War story: A team runs their balance controller as a high-priority thread on the same Jetson as their perception stack, isolated core, real-time priority, the works. It runs beautifully in the lab. In the field the robot occasionally staggers for no reason the control logs explain. The cause was memory-bandwidth contention: when the camera pipeline and a detection model hit peak bandwidth simultaneously, the control thread's memory reads stalled behind them, and a 1 ms loop occasionally ran 3 ms. The core was isolated; the memory bus was not. They moved the control loop to a separate MCU and the staggers vanished. Isolating a core does not isolate the memory system, and on a robot the memory system is the contended resource.

The lesson generalizes: isolation has to reach all the way down to the contended resource. Cores, caches, memory controllers, and buses are all shared, and the AI workload is the noisiest tenant on every one of them. When in doubt, put the deterministic loop on its own silicon.

Sizing compute: the roofline budget

Vendors sell accelerators on peak TOPS (trillions of operations per second). That number is close to useless for sizing, because most robot AI workloads never get near it. The reason is memory bandwidth, and the tool for reasoning about it is the roofline model.

Every kernel has an arithmetic intensity: the number of arithmetic operations it does per byte of memory it moves.

arithmetic_intensity  =  FLOPs / bytes_moved      (units: FLOP per byte)

A processor has two ceilings: peak compute (FLOP/s) and peak memory bandwidth (bytes/s). Which one you hit depends on the intensity:

attainable_FLOPs_per_s = min( peak_compute,
                              arithmetic_intensity * peak_bandwidth )

If a kernel's intensity is high (lots of math per byte, like a big dense matrix multiply with good reuse) you are compute-bound and TOPS matters. If it is low (little math per byte, like a depthwise convolution, an elementwise op, or anything memory-streaming) you are bandwidth-bound and TOPS is irrelevant, the memory system sets your speed. The crossover point, the ridge, is peak_compute / peak_bandwidth FLOP per byte. On typical edge accelerators the ridge sits high enough that a lot of real vision models land on the bandwidth side.

# Illustrative edge accelerator
# peak_compute   = 20 TFLOP/s   (FP16)
# peak_bandwidth = 100 GB/s
# ridge point    = 20e12 / 100e9 = 200 FLOP/byte
#
# A model at 40 FLOP/byte is bandwidth-bound: it can only reach
#   40 * 100e9 = 4 TFLOP/s, one fifth of the "20 TFLOP" sticker.
# To go faster you cut bytes moved (quantize), not add compute.

This is why quantization so often beats buying a bigger accelerator: it directly cuts the bytes moved, which is the thing you are actually limited by. It is also why two chips with identical TOPS can differ by 3x on the same model, the one with more memory bandwidth wins the bandwidth-bound layers.

The sizing procedure that actually works:

1. Profile the real model on the target chip, layer by layer, not the peak TOPS. Measure end-to-end latency and per-layer time.
2. Find the roofline position of the hot layers. Bandwidth-bound layers get faster with quantization and better memory layout; compute-bound layers get faster with more MACs or lower precision math.
3. Budget the whole latency chain (see the sensor section below). Inference is one link; capture, transfer, preprocess, and actuation are the others, and they often dominate.
4. Check sustained, not peak. Re-measure after ten minutes at thermal steady state. The throttled number is the real one.
5. Leave headroom. A processor run at 95% utilization has no margin for a heavier frame, a background task, or a hot day. Size for the peak workload at maybe 70% of sustained capacity.

Rule of thumb: If a model is bandwidth-bound (most edge vision is), the fastest win is fewer bytes: quantize, prune, use a smaller activation resolution, or improve memory layout. Adding compute you cannot feed does nothing.

Model optimization: quantization, pruning, distillation, compilation

A network trained on a workstation in FP32 is almost never what you deploy. The optimization toolkit exists to shrink it, in memory, bandwidth, and latency, until it fits the robot's budget, ideally with negligible accuracy loss. Four techniques carry most of the weight.

Quantization. Represent weights and activations in lower precision: FP16, INT8, or lower. INT8 is the workhorse. It quarters the memory footprint versus FP32, quarters the bandwidth (the thing you are usually limited by), and runs on dedicated integer units that are far more efficient than floating-point ones. The mechanism is an affine map from real values to integers:

q = round(r / scale) + zero_point          # quantize
r ≈ scale * (q - zero_point)                # dequantize
# scale and zero_point are chosen per-tensor or per-channel
# to cover the observed value range.

Two flavors. Post-training quantization (PTQ) takes a trained model and calibrates the scales on a small representative dataset, no retraining, minutes of work, typically under 1% accuracy loss for INT8 on well-behaved vision models. Quantization-aware training (QAT) simulates the quantization during training so the network learns to tolerate it, recovering accuracy on models that PTQ hurts, at the cost of a training run. Start with PTQ; reach for QAT only if PTQ costs too much accuracy. Per-channel scales (a separate scale per output channel) recover most of the accuracy that per-tensor quantization loses.

Pruning. Remove weights that contribute little. Unstructured pruning zeros individual weights and yields a sparse network that is smaller on disk but rarely faster on hardware, because general sparsity does not map to the dense units accelerators are built around. Structured pruning removes whole channels, filters, or attention heads, which shrinks the actual tensor dimensions and does speed up on any hardware. Structured pruning is what gives real robot latency wins. Some accelerators support a specific 2:4 structured sparsity pattern (two of every four weights zero) with hardware acceleration, which is the sweet spot when available.

Distillation. Train a small "student" network to mimic a large "teacher" by matching its outputs (soft labels) rather than only the ground-truth labels. The student learns the teacher's learned function with far fewer parameters, often reaching accuracy a same-size network trained from scratch cannot. This is how a large perception or policy model becomes something that fits on an NPU. It costs a training pipeline and produces a genuinely smaller model, with fewer parameters to store and fewer bytes to move, rather than a compressed copy of a large one.

Compilation. A graph compiler takes the trained model (usually via ONNX) and turns it into optimized code for the specific chip. It fuses adjacent layers (a convolution, a bias add, and a ReLU become one kernel, cutting memory round-trips), picks the best kernel implementation for the hardware, chooses tensor memory layouts, folds constants, and applies the chosen precision. TensorRT (NVIDIA), TVM, OpenVINO, and vendor NPU toolchains all do this. The speedup from compilation alone, before any quantization, is routinely 2-5x over a generic runtime, because the generic runtime runs each layer as a separate un-fused kernel with a memory round-trip between each.

Technique	Memory win	Latency win	Accuracy cost	Effort
INT8 PTQ	~4x	2-4x (integer units)	usually < 1%	Low (calibration set)
QAT	~4x	2-4x	near zero	Medium (retrain)
Structured pruning	model-dependent	scales with removed channels	tunable	Medium (retrain)
Distillation	large (smaller model)	large	design-dependent	High (train student)
Compilation	modest	2-5x	none (numerics aside)	Low (run the compiler)

The pipeline in practice: train in full precision, distill to a smaller architecture if you need to, prune structurally, quantize (PTQ first, QAT if needed), then compile for the target. Measure accuracy and latency at each step, because they compose in ways that are not always additive.

Rule of thumb: Do compilation and INT8 quantization first, they are cheap and give most of the win. Reach for pruning and distillation only when compile-plus-quantize still misses the budget. Always re-validate accuracy on the target hardware; a numerics difference between the workstation and the accelerator's kernels is a common and quiet source of accuracy drift.

Power and thermal budgets

Power and heat are the same problem seen twice, and on mobile robots they usually dominate the compute choice.

Start with the energy budget. Runtime is battery energy over total draw:

runtime_hours = battery_Wh / (P_compute + P_motors + P_sensors + P_overhead)

Compute is a controllable slice of the denominator. A 60 W compute module on a 500 Wh pack (a plausible small mobile robot) burns 12% of the energy budget continuously, cutting runtime by that much even when the robot is standing still, because the accelerator idles hot. A 10 W module doing the same job through better model optimization gives you back most of that runtime. See robot power & batteries for the pack side of this.

Now the thermal side, which is the constraint people forget. All that compute power comes out as heat, and the chip can only run at a given power if you can reject that heat and keep the junction below its throttle point. Steady-state junction temperature is set by the thermal path:

T_junction = T_ambient + P_dissipated * R_thermal
# R_thermal (deg C per watt) is the total resistance from
# silicon to air: die to heatsink to enclosure to ambient.

If a chip throttles at 95 C, the ambient is 40 C (a warm warehouse or a sun-baked outdoor robot), and your thermal resistance from junction to air is 3 C/W, then the sustainable power is (95 - 40) / 3 = 18 W, no matter what the datasheet says the peak is. A sealed enclosure with no airflow can have a thermal resistance several times worse, dropping the sustainable power further. This is why a module rated at 40 W frequently delivers 15-20 W of sustained work inside a real robot.

The design levers:

• Configurable power modes. Most robot compute modules let you cap power (a 15 W mode, a 30 W mode). Run the mode you can actually cool sustained, not the max. A capped mode with a good thermal path beats an uncapped mode that throttles chaotically.
• Thermal path. Conduction to the chassis (using the frame as a heatsink), heat pipes, and where acceptable, fans. Fanless designs trade sustained performance for silence and reliability, common near people and in dusty environments.
• Duty cycling. If the heavy workload is intermittent (a perception burst on demand rather than continuous), you can run above the sustained limit briefly and coast on thermal mass, as long as the average stays under budget.

War story: A drone-inspection team benchmarks their detection model at 45 fps on a compute module on the bench and specs the mission around it. In flight, in direct sun with the module in a sealed pod, it throttles to 18 fps within four minutes and the coverage math collapses. Nothing was wrong with the model or the chip. The bench had airflow and 22 C ambient; the pod had neither. They re-specced around the sustained 18 fps and added a conduction path to the airframe, which recovered part of it. Benchmark in the enclosure, in the ambient, at the duration the mission actually runs.

Sensor bandwidth and the latency chain

Compute does not run in a vacuum; it runs on a river of sensor data, and both the bandwidth of that river and the end-to-end latency from photons to actuation are design constraints in their own right.

Bandwidth. Sensors produce a lot of data. A single color camera at 1080p and 60 fps is on the order of hundreds of MB/s raw; a stereo pair or a multi-camera rig multiplies that; a spinning lidar adds a steady point-cloud stream; depth cameras add more. This data has to move over the interface (MIPI CSI-2 for cameras, Ethernet or USB for lidar and other sensors) into memory, and every byte competes for the same memory bandwidth your models need. It is common for the sensor ingest and preprocessing to consume as much memory bandwidth as the inference does. See robot sensors and machine vision for the sensor side.

The mitigations are about moving fewer bytes and moving them once: use hardware image signal processors and codecs that live next to the camera interface, keep data on the accelerator rather than round-tripping to CPU memory (zero-copy), and downsample early if the task allows.

Latency chain. The number that matters for a reactive robot is the whole chain from the world changing to the actuator responding, with inference as just one term in it:

t_total = t_capture + t_transfer + t_preprocess
          + t_inference + t_postprocess + t_actuate

# A reactive vision pipeline example (order of magnitude):
#   capture (exposure + readout)   ~ 10-30 ms
#   transfer to accelerator memory ~ 1-5 ms
#   preprocess (resize, normalize) ~ 1-5 ms
#   inference                      ~ 5-30 ms
#   postprocess (NMS, decode)      ~ 1-10 ms
#   actuation command out          ~ 1-5 ms
#   -------------------------------------------
#   total                          ~ 20-85 ms

Two lessons fall out. First, inference is often not the biggest term; camera exposure and readout, and the postprocessing (non-max suppression, decoding), frequently rival it. Optimizing only the model can leave most of the latency on the table. Second, this end-to-end latency sets how fast the robot can react and, for anything closing a control loop through perception, it directly limits the achievable loop gain, exactly the phase-lag problem that shows up in real-time control systems. A perception-in-the-loop system with 80 ms of latency cannot balance anything fast; that is why balance stays on a proprioceptive loop at 1 kHz and vision feeds slower guidance.

Rule of thumb: Optimize the whole latency chain and measure every stage. Capture and postprocessing are the terms people forget, and they are often the ones you can cut fastest.

ROS 2 on the edge

ROS 2 is the dominant middleware for the non-real-time part of the robot, the perception, planning, and coordination graph. Getting it to behave inside the compute and latency budget takes deliberate tuning, because the defaults are built for generality, not for a bandwidth-tight edge machine. See ROS 2 for the framework in depth.

The pressure points on the edge:

• Serialization and copies. By default, publishing a message serializes it and the middleware copies it, sometimes several times, across process boundaries. For a full-resolution image at 60 fps this is a bandwidth and latency killer. The fix is zero-copy / intra-process transport: when nodes share a process (composed nodes) or a shared-memory transport is configured, a large message is passed by pointer rather than copied. On an edge machine this is the difference between a camera pipeline that fits and one that saturates the memory bus. Type adaptation lets a message live on the GPU and be passed without a host round-trip, which matters enormously for vision.
• QoS tuning. ROS 2's quality-of-service settings (reliability, history depth, durability) directly set memory use and latency. A best-effort, keep-last-1 policy is right for a high-rate sensor stream where the freshest sample is all that matters and buffering stale frames wastes memory. Reliable delivery with deep history is right for commands you cannot drop. Wrong QoS is a common cause of edge memory bloat and latency spikes.
• Keep the hard-real-time loop out of the DDS graph. ROS 2's transport (DDS) is not hard-real-time. The control loop should live on the MCU or an isolated real-time thread and exchange only setpoints and state with the ROS 2 side over a deterministic channel. Putting a 1 kHz control loop inside the DDS graph invites jitter you cannot bound. This reinforces the split from earlier: ROS 2 is the best-effort brain; the deterministic loop is elsewhere.
• Executor and threading. The default single-threaded executor processes callbacks in one thread, which serializes work and can starve a time-sensitive callback behind a slow one. Multi-threaded and real-time executors, with callback groups to isolate the time-sensitive paths, keep the graph responsive under load on a constrained CPU.

The practical shape of a well-tuned edge ROS 2 system: composed nodes sharing a process for zero-copy on the heavy data path, QoS set per-topic to match the data's semantics, GPU-resident tensors passed by handle, and a hard boundary between the DDS graph and the real-time control running on separate silicon.

The cloud-edge division of labor

Not everything has to run on the robot. The question is what should, and the answer is set by latency tolerance, safety criticality, bandwidth, and connectivity reliability.

The dividing principle: anything on the critical path of not causing harm, and anything that must react within a control deadline, stays onboard. The network is neither fast enough (round-trips of tens to hundreds of milliseconds, worse on cellular) nor reliable enough (it drops) to be trusted with a safety function or a tight loop. A robot that stops working when the WiFi hiccups is not a product. So control, obstacle avoidance, balance, emergency stop, and the perception that feeds them are local, always.

What can offload is the slow, heavy, latency-tolerant, non-safety layer:

Layer	Where it runs	Why
Motor / current loop, safety	Onboard MCU	Hard deadline, safety-critical
Balance, reactive control	Onboard real-time core	Hard deadline
Perception feeding control	Onboard NPU/GPU	Latency and reliability critical
Local planning, SLAM	Onboard SoC	Needs to work offline, latency-sensitive
Global map building, fleet maps	Cloud	Heavy, shared across robots, latency-tolerant
Model training / fine-tuning	Cloud	Compute-heavy, offline, not on the robot
Model / policy updates (OTA)	Cloud to robot	Deployed periodically, not per-cycle
Heavy language reasoning, task planning	Cloud (optional)	Large models, seconds-tolerant, non-safety
Fleet telemetry, analytics	Cloud	Aggregation, no latency need

This maps onto the current split in learned high-level behavior. Big vision-language and vision-language-action models are heavy, and a robot may query a large cloud model for a slow, high-level decision ("what should I do with this scene") on a multi-second budget, while a small distilled policy runs the fast reactive control onboard. That two-speed structure, a slow deliberative layer that can be remote and a fast reactive layer that must be local, is the durable shape. See foundation models & VLAs for robotics for the model side of this.

The hybrid patterns worth knowing: offload with graceful degradation (use the cloud model when connected, fall back to a smaller onboard model when not, never stall), cloud for learning, edge for inference (robots run frozen models locally, the fleet's data trains better ones in the cloud, updates ship periodically), and cloud for the map, edge for the pose (a shared global map is built and stored centrally, each robot localizes against a local copy in real time).

Rule of thumb: If a task can tolerate seconds of latency, an occasional dropout, and is not safety-critical, it can live in the cloud. Everything else is onboard. When unsure, put it onboard; the failure mode of over-offloading is a robot that stops when the network does.

Representative hardware categories

Named by category rather than chasing specific part numbers that change yearly, the onboard-compute landscape sorts into a few families.

Motor-control and real-time MCUs. ARM Cortex-M and Cortex-R parts and motor-control DSPs, running the current loops and safety logic at milliwatts to a couple of watts, hard-real-time, on bare metal or an RTOS. This tier changes slowly because determinism, not throughput, is the spec.

Robotics SoC modules with integrated AI. The dominant category for the main brain: system-on-modules pairing multi-core ARM application processors with an integrated GPU and/or NPU, roughly 5-60 W, running Linux. NVIDIA's Jetson line (Orin-class and successors) is the reference point for GPU-heavy robot compute; Qualcomm's robotics platforms, NXP and TI robotics SoCs, and Apple-silicon-class parts fill out the space. Perception and learned policy run on the integrated accelerator, the ROS 2 graph on the CPU cores. Each generation adds NPU throughput per watt, which is what pulls larger models onboard.

Dedicated NPU / edge-AI accelerators. Fixed-function neural accelerators, sometimes a co-processor to a lighter host CPU, optimized for TOPS-per-watt on quantized models. Good when the workload is a well-defined perception model and efficiency beats flexibility.

FPGA and SoC-FPGA platforms. Reconfigurable logic (and hybrids with ARM cores next to FPGA fabric) for deterministic sensor processing, stereo depth, custom ISP, and hard-real-time control in high-volume or high-assurance products. Higher engineering cost, unmatched determinism.

x86 industrial and higher-power compute. For robots that can afford the power and heat (larger AMRs, autonomous vehicles, wheeled platforms with generous packs), an industrial x86 box or a discrete GPU brings workstation-class throughput. The exception on small mobile robots, the norm on vehicles, where the energy budget is measured in kWh. See self-driving cars & autonomous vehicles.

The durable pattern across all of them: a small deterministic tier for control, a Linux SoC with an integrated AI accelerator for perception and policy, and that accelerator sized to the model you must run within the power and thermal envelope you can actually cool.

Failure modes and outlook

The recurring failures are predictable once you know the budgets. Sizing to peak and running at sustained is the most common: specing around a fan-cooled bench benchmark, then watching the robot throttle in the field. Control jitter from a shared resource is next: a deterministic loop that shares a memory controller, cache, or bus with the AI workload eventually misses a deadline. Optimizing inference while capture and postprocessing dominate the chain wastes effort. Over-offloading to the cloud leaves a robot that stalls when the network drops. And a model quantized on the workstation but never re-validated on the accelerator's own kernels can lose accuracy quietly.

The direction of travel is clear enough to plan around. NPU throughput-per-watt improves each silicon generation, steadily pulling larger models (including compact vision-language-action models) onboard, so the cloud-edge line moves toward more local. Model optimization is maturing into a standard pipeline (compile, quantize, deploy) rather than a research project. Heterogeneous integration, application cores plus real-time cores plus NPU plus sometimes FPGA fabric on one module, is becoming the default, turning the control-versus-inference split into a matter of pinning work to the right block on one chip. The two-speed architecture, a slow deliberative model that may be remote and a fast reactive policy that is always local, looks durable because it follows from physics: the speed of light and the unreliability of networks keep the fast, safety-critical loop onboard. The chips get faster; the discipline stays the same.

Frequently asked questions

Can I run my whole robot on one processor? Rarely well. Hard-real-time control and AI inference have opposite requirements: control wants bounded worst-case latency, inference wants average throughput and achieves it with techniques that wreck worst-case latency. The standard architecture is at least a split between a deterministic tier (MCU or real-time core) for control and a Linux SoC with an accelerator for AI. On a single SoC you can approximate the split with a real-time core plus application cores, but a hard 1 kHz loop still belongs off the general-purpose Linux scheduler.

Do I trust the advertised TOPS number? No. TOPS is a peak that most robot workloads never reach, because they are limited by memory bandwidth rather than compute. Use the roofline model: profile your actual model on the target chip, find whether the hot layers are compute-bound or bandwidth-bound, and remember that most edge vision is bandwidth-bound, so memory bandwidth often predicts real speed better than TOPS does.

What is the single highest-leverage optimization? Compilation plus INT8 quantization, done together. Compilation fuses layers and picks the right kernels for the chip (routinely 2-5x on its own); INT8 quarters the memory footprint and bandwidth and runs on efficient integer units (another 2-4x), usually for under 1% accuracy loss with proper calibration. Both are cheap. Do them before you consider pruning, distillation, or buying a bigger chip.

Why does my model throttle in the robot but not on the bench? Thermal. The bench had airflow and cool ambient; the robot has a sealed or tight enclosure and warm ambient. Sustainable power is set by (throttle_temp - ambient) / thermal_resistance, and in a real enclosure that can be less than half the datasheet peak. The chip protects itself by dropping clocks. Benchmark in the actual enclosure, at the actual ambient, for the actual mission duration.

How do I keep the AI workload from disturbing my control loop? Isolate down to the contended resource. Isolating a CPU core is not enough, because the AI workload and the control loop still share the memory controller, cache, and bus, and a heavy model can starve the control thread of bandwidth. The robust answer is separate silicon: run the hard-real-time loop on an MCU or a dedicated real-time core with its own memory path, and exchange only setpoints and state with the AI side over a deterministic link.

When should I use an FPGA? When you need deterministic, high-bandwidth, fixed-function processing and can afford the development effort: stereo-depth computation, custom image signal processing, sensor pre-processing, or microsecond-deterministic motor control, especially at volume or in high-assurance products. For flexible, evolving AI models, a GPU or NPU is easier and usually the better choice; FPGAs win where the function is fixed and determinism is the spec.

How much can I offload to the cloud? Only the slow, heavy, latency-tolerant, non-safety layer: global map building, fleet learning, model updates, and optionally heavy high-level reasoning on a multi-second budget. Anything on the critical path of not causing harm, and anything with a control deadline, stays onboard, because the network is neither fast enough nor reliable enough to trust with a safety function. Design for graceful degradation so a dropout never stalls the robot.

Do I need a GPU on the robot at all? It depends on the workload. A proprioceptive control policy (a small MLP) runs in microseconds on a CPU core and needs no GPU. Vision, learned terrain perception, and vision-language-action models do need a GPU or NPU, because they are dense neural networks on image-scale inputs. Size the accelerator to the largest model you must run within the power and thermal budget, and no larger, since an idle oversized accelerator wastes energy continuously.

How does ROS 2 fit into an edge compute budget? As the middleware for the best-effort brain (perception, planning, coordination), tuned for the constrained machine: zero-copy or intra-process transport so large messages like images are passed by pointer rather than copied, per-topic QoS matched to the data's semantics to control memory and latency, and a hard boundary that keeps the hard-real-time control loop off the DDS graph and on separate deterministic silicon. Untuned ROS 2 on an edge machine drowns in serialization copies and QoS-driven buffering.