Depth Sensing: Stereo, ToF & Structured Light (The Ultimate Guide)

A robot that only sees pixels sees a flat world. It can name the objects in front of it and still drive its gripper straight through the table, because a classifier tells you what is in the frame and never how far. Depth is the missing coordinate, the one the robot's body actually lives in, and recovering it from an image sensor is a small family of tricks that trade against each other in ways that decide whether your perception stack works before you write a line of code.

This guide is about the three ways a camera-shaped device measures depth: passive and active stereo, which triangulate from the disparity between two views; structured light, which projects a known pattern and reads the distortion; and time-of-flight, which times the round trip of emitted light at every pixel, in both its indirect (phase) and direct (photon-timing) forms. These are the technologies inside every RealSense, ZED, Orbbec, Photoneo, and Azure Kinect on a robot today. We will work through the geometry that governs each, the specs that actually bind, where each one wins, how each one dies, and how they compare to LiDAR when the two overlap.

The take: the best depth camera is the method matched to your range, your lighting, and your accuracy budget. Stereo scales to range and survives sunlight because it is fundamentally a camera; structured light owns close-range sub-millimetre accuracy by averaging a known signal, and collapses the instant the sun or motion arrives; time-of-flight gives dense depth fast and cheap indoors and lies to you at corners and shiny surfaces. Pick by the one or two constraints that bind your application, because a sensor strong at everything except your binding spec is the wrong sensor.

Companion reading: LiDAR & depth cameras, machine vision, robot perception & pose estimation, robot sensors, and SLAM & localization.

Table of contents

1. Key takeaways
2. What depth sensing actually measures
3. Triangulation vs time-of-flight: the two physics
4. Passive stereo: disparity and the depth equation
5. Active stereo: painting texture with IR
6. Structured light: reading a known pattern
7. Time-of-flight: iToF and dToF
8. Method comparison
9. The specs that actually bind
10. Calibration and error sources
11. The hard cases: reflective, transparent, textureless, multipath
12. Depth cameras vs LiDAR
13. How to select
14. Frequently asked questions

What depth sensing actually measures

A depth camera produces a depth map: a per-pixel image where each value is the distance from the sensor to the nearest surface along that pixel's ray. Reproject that map through the camera intrinsics and you get a point cloud, a set of (x, y, z) points in the sensor frame. That geometry is what a planner needs to avoid an obstacle, what a grasp solver needs to place a gripper, and what a safety layer needs to trigger a protective stop at a real metric distance.

The reason depth deserves its own hardware category is that ordinary cameras throw it away. A lens maps the three-dimensional world onto a two-dimensional sensor, and that projection is not invertible: a small object up close and a large object far away land on the same pixels. A single image cannot tell them apart. Every depth technology in this guide is a way to break that ambiguity, either by adding a second viewpoint, by adding a known light source, or by timing light directly.

There is one axis that organizes the whole field: passive versus active. A passive sensor collects only ambient light, which makes it cheap, silent on the spectrum, and dependent on the scene giving it something to work with. An active sensor emits its own light and measures what returns, which lets it work in the dark and on blank surfaces at the cost of power, self-interference, and a losing fight against the sun. Passive stereo is the only fully passive method here. Structured light, time-of-flight, and active stereo all emit. Nearly every trade-off below follows from that split.

Triangulation vs time-of-flight: the two physics

Underneath the marketing, depth cameras run on one of two physical principles, and knowing which one you are holding predicts its error behaviour before you read the datasheet.

Triangulation recovers depth from geometry. Two viewpoints (two cameras, or a camera and a projector) separated by a known baseline see the same scene point along two different rays. The intersection of those rays fixes the point in space. Stereo and structured light are both triangulation. Their defining property is that depth resolution degrades with the square of distance, because the angular difference between the two rays shrinks as the point recedes, and eventually falls below what the sensor can resolve.

Time-of-flight recovers depth from the speed of light. Emit light, measure how long it takes to come back, multiply by c/2. There is no baseline and no triangulation, so depth is measured directly and its error does not explode with range. Instead it is limited by how precisely you can time the return, which is set by the signal-to-noise ratio of the reflected light. ToF cameras and flash LiDAR are the same physics in different packages.

Triangulation (stereo / structured light):
  Z = f·B / d          depth from disparity
  ΔZ ∝ Z² · Δd / (f·B)  error grows with Z²

Time-of-flight:
  Z = c·t / 2          direct, from round-trip time
  ΔZ roughly flat with range until SNR collapses

Rule of thumb: if you need accuracy that holds out to long range, you want the physics whose error does not scale with Z². If you need dense sub-millimetre depth up close and control the lighting, triangulation with a known pattern wins. The range where you operate picks the physics.

Passive stereo: disparity and the depth equation

Stereo is the most camera-like depth method, which is why robotics people reach for it first. It uses two ordinary image sensors, needs no emitter, scales to long range with a wider baseline, and works in sunlight.

Two cameras separated by a baseline B image the same world point at slightly different horizontal pixel positions. The difference in those positions is the disparity d. From similar triangles, depth is:

Stereo depth:   Z = (f · B) / d

  Z = depth (m)
  f = focal length (pixels)
  B = baseline (m)
  d = disparity (pixels)

Example: f = 700 px, B = 0.12 m
  d = 40 px  ->  Z = 700 · 0.12 / 40 = 2.10 m
  d = 10 px  ->  Z = 700 · 0.12 / 10 = 8.40 m
  d =  4 px  ->  Z = 700 · 0.12 /  4 = 21.0 m

Disparity falls off fast with distance. Far objects have tiny disparity, and once it drops below roughly one pixel you can no longer measure it. That is the stereo range ceiling, and it is why baseline matters so much.

Why error grows with the square of range

Differentiate the depth equation and you get the single most important fact about stereo:

Depth error:  ΔZ ≈ (Z² / (f · B)) · Δd

  Δd = disparity matching error (≈ 0.1-0.5 px for a good matcher)

Example: f = 700 px, B = 0.12 m, Δd = 0.2 px
  at Z = 2 m :  ΔZ ≈ (4   / 84) · 0.2 ≈ 0.010 m  (~1 cm)
  at Z = 8 m :  ΔZ ≈ (64  / 84) · 0.2 ≈ 0.152 m  (~15 cm)
  at Z = 20 m:  ΔZ ≈ (400 / 84) · 0.2 ≈ 0.95 m   (~1 m)

Depth error scales with Z². Go twice as far and error quadruples. This is geometry, not a defect you tune away. It dictates how you size the rig: to push usable range out you widen the baseline B or lengthen the focal length f (at the cost of field of view). A robot needing accurate depth at 15 m needs a wide-baseline rig, not a webcam-spaced pair.

The one runtime knob is Δd, the disparity matching error, and this is where the algorithm earns its keep. A block matcher that resolves disparity to the nearest whole pixel gives Δd ≈ 0.5 px. Fit a parabola to the matching-cost curve around its minimum and you recover the sub-pixel peak, dropping Δd toward 0.1 px and cutting depth error roughly fivefold. Semi-global matching (Hirschmuller's SGM, 2008) and its descendants win by producing smooth, confident, sub-pixel disparity fields. Learned stereo networks push Δd lower still on textured scenes. None of them escape the Z²/(f·B) scaling; they only shrink the constant out front.

Rule of thumb: stereo accuracy is set before runtime by baseline and focal length. However good your matcher, ΔZ ∝ Z²/(f·B). Choose the rig for the range you need, then optimize the matcher.

Rectification makes it real-time

Naively, finding the matching pixel in the second image is a 2D search across the whole frame. Rectification collapses it to 1D. Using the calibrated relative pose of the two cameras, you warp both images so that corresponding points always lie on the same horizontal row. Now the match is a search along a single scanline, which is what makes dense stereo tractable at video rate. Rectification depends entirely on calibration being correct, which is the theme of the calibration section below.

Active stereo: painting texture with IR

Passive stereo has one fatal dependency: it needs texture. The matcher correlates local image patches between the two views, and a blank white wall, a glossy panel, or a dim corridor gives it nothing to correlate. Depth comes back full of holes.

Active stereo fixes this by adding an infrared projector that sprays a static, semi-random dot pattern onto the scene. The key detail is that the matcher uses the extra contrast only to find correspondences, and decoding the pattern is structured light's job. This means the projected pattern does not need to be known or calibrated with any precision, which makes active stereo robust and cheap. The Intel RealSense D400 series is the canonical example: it works in the dark, on blank walls, because the projector paints texture, and it still works in bright sunlight because when the projector is washed out there is usually enough natural texture to fall back to passive matching.

That graceful degradation across lighting is why active stereo is the most versatile depth camera family for robots that move between indoors and outdoors. It never gives sub-millimetre accuracy the way structured light does, and it still carries stereo's Z² error growth, but it rarely returns nothing.

War story: a mobile manipulator kept failing to detect a white shipping crate against a white wall. The depth image was a clean hole exactly where the crate stood. The passive stereo matcher had no texture to lock onto on either surface. Switching to a unit with an active IR projector filled the hole instantly; the dots gave the matcher the contrast the paint had denied it. The geometry stayed the same; the only change was whether the scene offered anything to correlate.

Structured light: reading a known pattern

Structured light also triangulates, but it replaces one of the two cameras with a projector that throws a known, coded pattern: stripes, a pseudo-random dot cloud, or a temporal sequence of patterns. Because the pattern is known, a single identified feature gives an absolute, high-precision depth with none of the matching ambiguity that limits passive stereo. This is why structured light owns the close-range accuracy crown.

How it reaches sub-millimetre

The workhorse is phase-shifting profilometry. Project N sinusoidal fringe patterns, each shifted by 2π/N, and recover the per-pixel phase in closed form from the intensity samples:

N-step phase shift:
  φ(x,y) = atan2( Σ Iₙ·sin(2πn/N) , Σ Iₙ·cos(2πn/N) )

Because that phase is estimated from N intensity measurements, its noise falls as 1/sqrt(N), and the estimate has sub-pixel resolution independent of the projector's pixel pitch. A coarse Gray-code sequence unwraps the absolute fringe order so a smooth phase becomes an absolute coordinate. Stack those and you reach sub-millimetre depth precision at 0.3-1 m. This is why industrial 3D scanners and high-end bin-picking sensors (Photoneo PhoXi, Zivid) are structured-light: when the task is finding a 2 mm chamfer on a part in a bin, nothing else is this precise.

The accuracy comes from averaging a known signal over multiple frames, and that is also the weakness. The averaging assumes a static scene and a dark room, and it surrenders both the moment the part moves or the sun comes up.

Single-shot vs multi-shot

Multi-shot (temporal coding, the phase-shift and Gray-code sequences above) is the most accurate and needs a static scene during capture: any motion smears the code. Single-shot decodes depth from one spatially coded frame, like the original Kinect v1's fixed dot cloud. It tolerates motion and runs at video rate but is markedly less precise. Choose by whether your scene holds still. A scanner over a static parts bin can multi-shot; a sensor over a moving conveyor must single-shot.

Why it dies in sunlight

The projected pattern is a few milliwatts of IR. Direct sunlight delivers roughly 1000 W/m² across the spectrum, a large chunk of it in the near-IR band the sensor uses. The sun overwhelms the pattern's contrast: the camera sees sun-flooded pixels, the code is unreadable, and depth collapses. No coding scheme closes a four-orders-of-magnitude irradiance gap. Structured light is an indoor technology, full stop. It also degrades when multiple units share a space, because their patterns interfere, unless they are time-multiplexed or use distinct codes.

Time-of-flight: iToF and dToF

A time-of-flight camera puts a flash-LiDAR principle into a camera body: an IR emitter floods the whole scene and a specialized 2D sensor measures the round trip at every pixel at once. Depth is measured directly rather than triangulated, so it does not blow up with Z², and the result is a dense depth image at high frame rate with low latency. There are two ways to do it.

Indirect ToF (iToF)

iToF modulates the emitter as a continuous sine wave and measures the phase shift between emitted and received light at each pixel. Distance follows from the phase:

iToF range from phase:  R = (c / (4π·f_mod)) · φ

  f_mod = modulation frequency
  φ     = measured phase shift (radians)

Unambiguous range:  R_max = c / (2 · f_mod)
  f_mod = 20 MHz  ->  R_max = 7.5 m
  f_mod = 100 MHz ->  R_max = 1.5 m

Higher modulation frequency buys precision and shrinks the unambiguous range: past R_max the phase wraps, and a 9 m target reads as 1.5 m. The precision link is direct, σ_R ≈ (c/(4π·f_mod))·σ_φ with σ_φ ∝ 1/SNR, so doubling f_mod halves depth noise and halves R_max. You cannot get both from one frequency, which is why serious iToF sensors run multi-frequency capture (say 20 MHz plus 80 MHz): the high frequency sets precision, the beat between frequencies sets the unambiguous range through a Chinese-remainder-style unwrap. iToF is the mainstream camera approach; the Microsoft Azure Kinect and Orbbec Femto are iToF, giving good resolution and precision indoors from 0.5-5 m.

Direct ToF (dToF)

dToF times individual photons with SPAD (single-photon avalanche diode) arrays, exactly like dToF LiDAR. It builds an arrival-time histogram over many pulses and picks the peak, sharpening as 1/sqrt(N) in the accumulated pulses. dToF is more robust to multipath and ambient light and scales to longer range, historically at lower pixel resolution than iToF. It is the technology in phone LiDAR sensors and a growing share of automotive flash units. As SPAD pixel counts climb, the resolution gap is closing.

Motion and the frame budget

Frame-to-depth budget at 30 fps:
  per-frame time = 1/30 s ≈ 33 ms
  iToF captures several phase sub-frames within that window
  -> fast motion during the 33 ms smears depth ("motion blur" in Z)

Because iToF stacks multiple sub-frames per depth frame, a fast-moving object smears in Z, producing edge artifacts at moving boundaries. dToF, timing per shot, tolerates motion better. Both suffer at depth discontinuities where a pixel straddles a near and a far surface and averages them into a flying pixel floating in the gap.

Method comparison

The three methods (treating passive and active stereo as one family) line up as follows. Read it as a map of where each one wins and where it collapses, because no row is a small effect.

Property	Stereo (passive / active)	Structured light	Time-of-flight (iToF / dToF)
Physics	Triangulation from disparity	Triangulation from pattern	Round-trip time or phase
Active light?	Optional (active stereo)	Yes (IR pattern)	Yes (IR flood)
Depth error vs range	Grows as Z²	Grows as Z²	Roughly flat until SNR fails
Close-range accuracy	Good	Excellent (sub-mm to mm)	Good
Long-range scaling	Best (widen baseline)	Poor (pattern fades)	Moderate
Sunlight outdoors	Works (passive especially)	Fails	Degrades badly
Featureless surfaces	Fails (passive), OK (active)	Works	Works
Frame rate	High (limited by matching)	Low (multi-shot) to moderate	High
Resolution	High (= camera sensor)	High	Lower (sensor-limited)
Multipath / scattering	Not affected	Some	Yes (its worst flaw)
Minimum range	Wide-baseline loses near objects	0.3-0.4 m typical	0.2-0.3 m typical
Relative cost	$-$$	$$$-$$$$	$$
Typical robotics use	AMR, outdoor, general pick	Bin-picking, inspection, scanning	Indoor mapping, people, gestures
Example products	RealSense D455, ZED 2i, OAK-D	Photoneo PhoXi, Zivid, Kinect v1	Azure Kinect, Orbbec Femto

The one-line summary: stereo for outdoors and range, structured light for close-range accuracy, time-of-flight for fast dense indoor depth. The rest of this guide is why each is true and where each breaks.

The specs that actually bind

Datasheets are written to flatter. Here is the engineer's checklist for a depth camera, and what to distrust on each line.

Range, and at what reflectivity

Maximum range is meaningless without a target reflectivity. A depth camera rated to 6 m usually means against a favourable, textured, matte surface. Against a dark (10% reflective) or glossy target the usable range can halve. For structured light and ToF the range ceiling is the technology: structured light to roughly 2-5 m, ToF to roughly 5-8 m, stereo to whatever the baseline supports.

Accuracy vs precision, versus distance

These are different and both matter. Accuracy is how close the mean measurement sits to truth (bias). Precision (repeatability) is the scatter of repeated measurements (noise). A sensor can be precise but biased (a consistent 3 cm offset, correctable by calibration) or accurate but noisy (right on average, useless per frame). Both degrade with distance, for stereo and structured light as Z², for ToF more gently. Demand the curve, not a single headline number.

Minimum range, the forgotten spec

Every active sensor has a blind zone up close where the return saturates or the geometry breaks. Structured-light and ToF units often cannot measure inside 0.2-0.3 m. A wide-baseline stereo rig loses near objects because they fall outside one or both frustums, or their disparity exceeds the search range. For a wrist-mounted manipulation camera, minimum range is frequently the binding constraint, because you cannot grasp what is too close to see.

Field of view and depth resolution

Horizontal by vertical FoV sets how much of the world you capture per frame, and it trades against angular resolution and range because a wider spread thins the returning energy per pixel. Depth-map resolution (640×480, 1280×720, 1024×1024) sets the smallest feature you can resolve at a given range. Match resolution to the smallest feature you must detect at your working distance, then stop, because every extra pixel is extra compute.

Frame rate and latency

ToF and single-shot stereo run at 30-90 fps with low latency, which is why they win for people-tracking and closed-loop visual servoing. Multi-shot structured light captures several frames per depth map, so its effective rate is low and it needs a static scene. If your target moves, frame rate and motion tolerance bind harder than accuracy.

Sunlight performance

The great divider, and it deserves its own line because it eliminates candidates outright. Passive stereo works and often prefers sun. Active stereo degrades gracefully. ToF degrades badly as the IR background eats dynamic range. Structured light fails. If any part of the robot's life is outdoors in daylight, this row removes half the field before you read anything else.

Power and thermal

USB depth cameras draw roughly 1-5 W, but the IR projector and the on-board depth ASIC add heat inside a sealed enclosure. On a battery robot that power is a real fraction of the budget, and thermal throttling of a depth ASIC in a hot enclosure is a classic field failure that looks like the sensor "getting noisy" as it warms up.

Rule of thumb: pick the one or two specs that bind your application, often minimum range and sunlight for manipulators, range-at-10% and accuracy-vs-distance for mobile, and treat the rest as tie-breakers. A sensor strong everywhere except your binding spec is the wrong sensor.

Calibration and error sources

Depth cameras live and die on calibration, and triangulation methods most of all.

Intrinsics and extrinsics. Each camera has intrinsics (focal length, principal point, lens distortion) and the pair has extrinsics (the exact relative pose). Stereo rectification, and therefore every disparity, depends on both being correct. A rig knocked out of calibration by a thermal cycle or a bump produces depth that is confidently, smoothly wrong: no holes, no obvious error, just a systematic bias that poisons everything downstream. This is why factory-calibrated, rigid-baseline modules (RealSense, ZED, OAK-D) exist. Two loose cameras you hand-calibrate will drift, and you will chase that drift forever.

Temperature. The baseline is a physical distance between two lenses, and it changes as the housing expands. A few tens of microns of baseline shift is enough to bias depth at range. Good modules use a rigid, low-expansion frame and sometimes an online self-calibration that re-estimates extrinsics from the scene.

Systematic ToF errors. iToF carries a wiggling error (a periodic bias from the emitted waveform not being a perfect sinusoid) and a temperature-dependent phase offset, both handled by per-unit calibration tables. Skip them and a "calibrated" ToF sensor still reads a centimetre off in a repeatable pattern.

The hand-eye transform. For a camera on a robot, depth in the sensor frame is useless until you know the sensor's pose relative to the robot base or the gripper. That hand-eye calibration (eye-in-hand or eye-to-hand) is its own procedure, and a 1° or 2 cm error in it becomes a systematic error in every grasp pose, indistinguishable from sensor noise until you check it. See robot calibration for the full treatment.

Rule of thumb: when depth is smoothly wrong rather than holey, suspect calibration or temperature, not the transducer. Holes are a signal problem; biases are a geometry problem.

The hard cases: reflective, transparent, textureless, multipath

Every depth technology has a set of surfaces and geometries that break it, and they are the surfaces you meet in the real world.

Textureless surfaces. A blank wall, a matte white panel, a clear sky. Passive stereo returns holes because there is nothing to match. Active stereo, structured light, and ToF handle these fine, because they supply their own signal. This is the single strongest argument for an active method indoors.

Specular and reflective surfaces. A glossy floor, brushed metal, a mirror. The emitted light reflects away from the sensor instead of scattering back, so you get holes, or it reflects the sensor's own signal from somewhere else, so you get a phantom surface behind the mirror. All active methods struggle. Passive stereo can sometimes match the reflected image and place depth at the mirror world, which is its own kind of wrong.

Transparent objects. Glass, clear plastic, a bottle of water. Light passes through, so the sensor measures the surface behind the object, and the transparent object is invisible in the depth map. This defeats every optical depth method and is an open research problem; the practical fixes are polarization cues, learned priors, or a different modality (tactile or capacitive sensing) for the final approach.

Multipath, the ToF signature failure. Emitted light reaches a surface directly and also arrives late after bouncing off other surfaces, and the late arrivals corrupt the per-pixel phase or time. The textbook case is a concave corner: light bounces wall-to-wall before returning, and the corner reads rounded or pushed back. Shiny floors, retroreflectors, and translucent objects produce related errors. Multipath is intrinsic to flood illumination, which is why a structured-light or stereo sensor can beat a ToF sensor on a geometrically tricky scene even when the ToF sensor has better nominal precision. Mitigations are multi-frequency capture, multipath-aware processing, and dToF's better separation of direct from indirect returns.

Depth discontinuities and flying pixels. At an object edge, a single pixel images both the near object and the far background, and the sensor averages them into a point floating in the gap. Every method shows this to some degree. The fixes are edge-aware filtering, confidence thresholds, and for stereo, left-right consistency checks that discard pixels whose forward and reverse matches disagree.

Depth cameras vs LiDAR

Depth cameras and LiDAR overlap, and the overlap is where most sensor-choice mistakes happen. The clean division:

Depth cameras win at short range indoors, on density, and on cost. A depth camera returns a full dense depth image (hundreds of thousands of points) covering a frustum at 30-90 fps for a few hundred dollars and a few watts. LiDAR returns a sparser cloud, a set of scan lines with gaps between them, and costs and weighs more. For manipulation at 0.3-1.5 m, for reading the shape of an object, for people-tracking indoors, a depth camera is the right tool.

LiDAR wins outdoors, at range, in sun, and where you need lighting-independent geometry. LiDAR times its own light, so it is far more robust to surface reflectivity and, at 1550 nm or with FMCW coherent detection, largely immune to sunlight. Its range reaches 100-250 m where a depth camera runs out at 5-20 m, and a spinning unit gives full 360° coverage no camera frustum matches. For an outdoor mobile robot or a vehicle, LiDAR carries the long and wide picture.

The Z² versus flat-error contrast is the crisp version: triangulating depth cameras lose accuracy quadratically with range, while LiDAR (direct time-of-flight) holds accuracy roughly flat until its photon budget runs out. That is exactly why depth cameras own the near field and LiDAR owns the far field.

Most capable robots run both. An indoor AMR pairs a 2D LiDAR for navigation with a forward depth camera to catch overhangs the scan plane misses. A humanoid carries head depth cameras for manipulation and a LiDAR or camera ring for locomotion awareness. The two fuse so each covers the other's blind spots. The LiDAR and depth cameras guide goes deep on the LiDAR side and the fusion patterns; the SLAM and localization guide covers what consumes the fused geometry.

How to select

Choose in this order, because each criterion eliminates candidates before the next matters: range → lighting → accuracy → field of view → budget → integration.

1. Range and minimum range. Close indoor work (0.3-2 m) points to a depth camera; check the minimum range against your closest target, because that is what usually bites a manipulator. Beyond about 10 m or outdoors, you are into LiDAR territory.
2. Lighting. Any direct sun eliminates structured light immediately and pushes you to passive or active stereo. Dark or featureless indoor scenes eliminate passive stereo and favour active stereo, ToF, or structured light.
3. Accuracy. Sub-millimetre for inspection or bin-picking means structured light. Centimetres for navigation means almost anything, but remember stereo's Z² error growth when you size the rig.
4. Field of view. A wide frustum for obstacle awareness trades against angular resolution and range. Match FoV to the task rather than buying the widest number.
5. Budget and power. Active stereo and ToF are cheap and low-power. High-end structured-light scanners cost an order of magnitude more and are worth it only when their accuracy is the binding spec.
6. Integration. A stable ROS 2 driver, correct per-point timestamps, and a solid TF transform to the robot base are worth more than five percent on any spec line. The hardware rarely fails; the integration usually does.

Rule of thumb: for a manipulator, minimum range and accuracy bind. For a mobile robot, sunlight and range-at-low-reflectivity bind. Name your two binding specs first, then let them eliminate the field before you compare anything else.

For robot-class-specific picks, a warehouse AMR pairs a depth camera with a 2D safety scanner; a picking cell uses eye-in-hand active stereo for general parts and structured light for small or shiny ones; an outdoor platform leans on active stereo (for its sun tolerance) plus LiDAR. The perception that runs on the resulting depth (segmentation, pose estimation, grasp synthesis) is the bridge back to the machine vision and robot perception & pose estimation guides.

Frequently asked questions

Which depth technology should I start with for a general indoor robot? Active stereo (RealSense D400 class, OAK-D, or a ZED) is the safe default. It works in the dark and on blank walls thanks to the IR projector, degrades gracefully in sunlight by falling back to passive matching, covers roughly 0.3-6 m, and is cheap and well supported in ROS 2. You reach past it only when a specific spec binds: sub-millimetre accuracy (go structured light) or long outdoor range (add LiDAR).

Why does my depth image have holes? Holes mean the sensor got no usable measurement for those pixels. For passive stereo it is lack of texture (blank walls, glossy surfaces). For structured light or ToF it is sun saturation, an out-of-range surface, a specular reflection bouncing the light away, or a black absorptive material that returns too few photons. Active IR projection, lighting control, or a different technology fixes most of it. Holes are a signal problem, distinct from smooth bias, which is a calibration problem.

Why does my ToF camera read corners as rounded or pushed back? Multipath. Light bounces between the two walls of the corner and arrives late, corrupting the per-pixel phase or time measurement. It is intrinsic to flood-illuminated ToF. Mitigations are multi-frequency capture, multipath-aware processing, or switching to structured light or stereo for geometrically tricky scenes.

Can stereo or structured light work outdoors? Passive stereo works and often prefers sunlight, because the sun provides the texture it needs to match. Active stereo works, falling back to passive matching when the IR projector is washed out. Structured light does not work outdoors: direct sun at roughly 1000 W/m² overwhelms the milliwatt projected pattern. ToF is degraded outdoors but sometimes usable in shade.

How far can a stereo camera actually measure? It depends entirely on baseline B and focal length f, because Z = f·B/d and error grows as Z². A 95-120 mm baseline module is usable to roughly 6-20 m before error becomes unacceptable; survey rigs with metre-class baselines reach much further. There is no fixed answer. Compute ΔZ ≈ Z²·Δd/(f·B) for your rig and your accuracy tolerance and read off the range where it crosses your limit.

Structured light vs ToF for indoor mapping: which is better? ToF, in most cases. It gives dense depth at 30 fps with low latency and tolerates a moving sensor, which is what mapping and people-tracking need. Structured light is more accurate but its multi-shot modes need a static scene, so it belongs on a fixed inspection or bin-picking station, not on a moving robot building a map.

What sensor goes on a robot arm for picking? A depth camera, mounted eye-in-hand (on the wrist) or eye-to-hand (fixed overhead). For precision bin-picking of small or shiny parts, structured light (Photoneo, Zivid). For general pick-and-place, active stereo or ToF is faster and cheaper. The binding specs are usually minimum range and the hand-eye calibration, not maximum range. See robot calibration.

How do I deal with transparent or mirror-like objects? No optical depth method sees clear glass or a mirror reliably; the light passes through or reflects away. Practical options are polarization-based sensing, learned depth-completion priors trained on transparent objects, staging the scene to avoid the geometry, or switching to a contact or capacitive sensor for the final approach. Treat transparent-object grasping as an open problem, not a sensor you can buy your way out of.

Do I need LiDAR if I already have a depth camera? Often not, indoors and at short range: a good active-stereo or ToF camera covers 0.3-6 m densely and cheaply. You need LiDAR when you go outdoors in sun, need range beyond about 10 m, need 360° coverage, or need lighting-independent geometry for robust SLAM. Many robots run both, LiDAR for the long and wide picture and a depth camera for the close and dense one.

Why do my depth measurements drift as the sensor warms up? Temperature. The stereo baseline is a physical distance between two lenses that expands with heat, and ToF carries a temperature-dependent phase offset. Both bias depth in a repeatable way. Good modules use a rigid low-expansion frame, per-unit thermal calibration tables, and sometimes online self-calibration. If depth is smoothly wrong and changes with runtime, suspect thermal drift before the transducer.