EmbodiedEdge Evidence

Bench2Field · Case study 07

Alpamayo 2 Edge with TensorRT Edge-LLM on the Jetson AGX Thor

NVIDIA's deployment recipe replicated on the Jetson board and pushed further · JetPack 7.1-b112, TensorRT 10.13.3.9, CUDA 13.0, MAXN · 12 November 2026
Reasoning tokens per second: 93 tokens / s, 10.8 ms per token, greedy050100150NVFP4: 66.6489 tokens / sNVFP493tokens / sReasoning tokens per second10.8 ms per token, greedy
Decisions per second: 2.7 Hz, median decision, 367 ms median, 427 ms p95, FP8 recipe, nine clipsmeets the 10 Hz planner deadline024681012PyTorch: 3.0175 Hz, median decisionPyTorchNVIDIA: 4.54545 Hz, median decision, published, not run hereNVIDIAEmbedl: 10.8814 Hz, median decision, published, not run hereEmbedl2.7Hz, median decisionDecisions per second367 ms median, 427 ms p95, FP8 recipe, nine clips
Energy per decision: 29 joules, 85 W mean while deciding, MAXN01020304050Embedl: 12.45 joules, published, not run hereEmbedlNVFP4: 37.18 joulesNVFP429joulesEnergy per decision85 W mean while deciding, MAXN
Deadline missed367 ms against the 100 ms planner budget
Same chain of causationNVIDIA's example sentence, word for word
Tail improvedp95 544 to 427 ms against the PyTorch stream
NVFP4 slower462 ms median with the language model in W4A16
Drafter not releasedthe recipe's 220 ms needs a DFlash2 drafter
7 issues loggedeach with its cause and fix, in the friction log
needle, measured heregrey marker, published figure, not run heremarker in text colour, another build measured heregreen band, deadline met

Where every build lands

Per decision, back to back with the engines loaded once, five Euler steps, the six newest camera frames encoded per decision (the recipe's rolling mode), MAXN. Nine clips, 20 decisions each: the eight of case studies 02 and 06 and NVIDIA's example clip. The PyTorch row is case study 06's 10 Hz stream on the same board (30 decisions). Hatched rows are published figures, not measured here.

medianp95 (Embedl reports p90)published, not run here100 ms deadline
Median and tail latency per decision, by build0200400600msPyTorch stream, 10 Nov (case study 06)PyTorch stream, 10 Nov (case study 06): median 331.4 ms331PyTorch stream, 10 Nov (case study 06): p95 543.9 ms544 p95FP8 recipe as writtenFP8 recipe as written: median 367.2 ms367FP8 recipe as written: p95 427.2 ms427 p95FP8, three Euler stepsFP8, three Euler steps: median 352.6 ms353FP8, three Euler steps: p95 511.7 ms512 p95FP8, six candidatesFP8, six candidates: median 413.1 ms413FP8, six candidates: p95 493.3 ms493 p95NVFP4 W4A16 language modelNVFP4 W4A16 language model: median 461.5 ms462NVFP4 W4A16 language model: p95 593.1 ms593 p95NVFP4 W4A16, six candidatesNVFP4 W4A16, six candidates: median 511.9 ms512NVFP4 W4A16, six candidates: p95 627.3 ms627 p95NVIDIA, DRIVE AGX Thor, with drafterNVIDIA, DRIVE AGX Thor, with drafter: median 220.0 ms220published, not run hereEmbedl card, same boardEmbedl card, same board: median 91.9 ms92Embedl card, same board: p90 140.3 ms140 p90published, not run here100 ms deadline (10 Hz)
buildmedian, msp95, msJ per decisionmean Wrecord
PyTorch stream, 10 Nov (case study 06)331544––06_alpamayo2_edge/runs/stream_bench-thor_tuned_fp8_steps5_maxn.json
FP8 recipe as written36742729.185runs/stream/clips9_rolling_c1.json
FP8, three Euler steps35351229.486runs/stream/clips9_rolling_c1_s3.json
FP8, six candidates41349333.484runs/stream/clips9_rolling_c6.json
NVFP4 W4A16 language model46259337.284runs/stream/clips9_rolling_c1_nvfp4-w4a16.json
NVFP4 W4A16, six candidates51262742.185runs/stream/clips9_rolling_c6_nvfp4-w4a16.json
NVIDIA, DRIVE AGX Thor, with drafter220–––published: indicative, recipe README; range 180 to 250 ms over ten clips
Embedl card, same board92140 (p90)12.4–published: model card, 2026-11-11: NVFP4 language model, MXFP8 vision and expert, caches, speculation; one clip, 180 decisions

Where a decision's time goes

Stage medians, stream-synchronized at each boundary as the recipe's own runner does. The chain of causation is greedy decoding, one token at a time; the recipe's speculative drafter is not released for the Edge model.

vision encoderprefillchain of causationaction expertexpert K/V copy
Stage cost per decision, median, by build0200400ms, sum of stage mediansFP8 recipe, one candidate: vision encoder 24.7 msFP8 recipe, one candidate: prefill 94.7 ms95FP8 recipe, one candidate: chain of causation 207.1 ms207FP8 recipe, one candidate: action expert 34.5 msFP8 recipe, one candidate: expert K/V copy 1.1 ms362 msFP8 recipe, onecandidateFP8, three steps: vision encoder 24.7 msFP8, three steps: prefill 93.9 ms94FP8, three steps: chain of causation 208.3 ms208FP8, three steps: action expert 25.3 msFP8, three steps: expert K/V copy 1.1 ms353 msFP8, three stepsFP8, six candidates: vision encoder 24.8 msFP8, six candidates: prefill 94.9 ms95FP8, six candidates: chain of causation 208.7 ms209FP8, six candidates: action expert 83.2 ms83FP8, six candidates: expert K/V copy 2.9 ms414 msFP8, sixcandidatesNVFP4 W4A16, one candidate: vision encoder 33.1 msNVFP4 W4A16, one candidate: prefill 124.0 ms124NVFP4 W4A16, one candidate: chain of causation 270.7 ms271NVFP4 W4A16, one candidate: action expert 35.3 msNVFP4 W4A16, one candidate: expert K/V copy 1.3 ms464 msNVFP4 W4A16, onecandidateFP8, cold vision cache, example clip: vision encoder 97.8 ms98FP8, cold vision cache, example clip: prefill 99.0 ms99FP8, cold vision cache, example clip: chain of causation 242.0 ms242FP8, cold vision cache, example clip: action expert 85.0 ms85FP8, cold vision cache, example clip: expert K/V copy 2.8 ms527 msFP8, cold visioncache, exampleclip

With no reasoning at all the floor is vision, prefill and expert: about 155 ms at one candidate, already above Embedl's published median. Each token of reasoning costs 10.8 ms of GPU work (measured with the engine alone, with and without a host sync per token), so a 22-token sentence is about 237 ms of the decision. The NVFP4 engine holds 1.62 GB of packed weights against 1.98 GB in FP8, yet its token costs 15.0 ms on the example clip against 11.0 in FP8: on Edge-LLM 0.11.1 with TensorRT 10.13.3.9 the single-token NVFP4-A16 kernel does not turn halved bytes into time, and W4A16 adds dequantization work to the compute-bound prefill.

The nine clips

FP8 recipe, one candidate, median of 20 decisions per clip. Four clips take the model's no-reasoning path (<NOMINAL>, 6 tokens) and finish in about 220 ms; the five that write a sentence take 367 to 427 ms.

writes a chain of causationno reasoning (nominal path)
Median latency per clip, FP8 recipe, one candidate0200400600ms, median of 20 decisions030c760c, following a slowing lead vehicle030c760c: 427.2 ms, 26 tokens427 ms, 26 tokens06b483cf, straight through an intersection06b483cf: 395.4 ms, 22 tokens395 ms, 22 tokensea7bbd31, NVIDIA's example, right turnea7bbd31: 393.5 ms, 22 tokens394 ms, 22 tokens174407b0, all-way stop174407b0: 380.7 ms, 21 tokens381 ms, 21 tokens09ad74a2, straight, green light09ad74a2: 367.2 ms, 19 tokens367 ms, 19 tokens0ea6fd880ea6fd88: 221.7 ms, 6 tokens222 ms, no reasoning0a1ef8080a1ef808: 221.3 ms, 6 tokens221 ms, no reasoning195f1fc6, left turn195f1fc6: 220.5 ms, 6 tokens220 ms, no reasoning0347d9f9, right turn0347d9f9: 217.8 ms, 6 tokens218 ms, no reasoning100 ms deadline
clipmedian, mstokenschain of causation (FP8; NVFP4 identical unless noted)
030c760c, following a slowing lead vehicle42726Decelerate to maintain a safe following distance behind the slowing lead vehicle and prepare to stop for the red traffic light ahead. NVFP4 differs: "Decelerate to maintain a safe following distance behind the lead vehicle as it slows for the cyclist and traffic queue ahead."
06b483cf, straight through an intersection39522Proceed straight through the intersection to follow our route, as the roadway and crosswalk are clear.
ea7bbd31, NVIDIA's example, right turn39422Decelerate and turn right at the intersection to follow our route, as the roadway is clear. NVFP4 differs: "Decelerate and turn right at the intersection to follow our route, as the roadway is clear of conflicting traffic."
174407b0, all-way stop38121Decelerate to stop at the stop line for the all-way stop sign at the upcoming intersection.
09ad74a2, straight, green light36719Proceed straight through the intersection to follow our route, as the traffic signal is green. NVFP4 differs: "Proceed straight through the intersection to follow our route, as the traffic light is green."
0ea6fd882226<NOMINAL>
0a1ef8082216<NOMINAL>
195f1fc6, left turn2206<NOMINAL>
0347d9f9, right turn2186<NOMINAL>

Did the faster builds still drive the same way?

Trajectories decoded with the recipe's host/compare.py from the action expert's output, ADE in metres over the 64 waypoints, same noise seed across builds. The grey bar is the spread among the six candidates the FP8 engines sample for the same clip (mean marked), the study's budget: a build whose difference sits inside it changes the plan less than the model's own sampling does.

FP8 against NVFP4 W4A16five against three Euler stepsspread among the six FP8 candidates, max; mean marked
Trajectory difference between builds, per clip, against the candidate spread01234ADE, metres over 64 points030c760c030c760c: spread among the six FP8 candidates, mean 0.76 m, max 1.95 m030c760c: FP8 against NVFP4 W4A16, 0.008 m030c760c: five against three Euler steps, 0.035 m0347d9f90347d9f9: spread among the six FP8 candidates, mean 1.66 m, max 3.58 m0347d9f9: FP8 against NVFP4 W4A16, 0.329 m0347d9f9: five against three Euler steps, 0.187 m06b483cf06b483cf: spread among the six FP8 candidates, mean 1.55 m, max 4.14 m06b483cf: FP8 against NVFP4 W4A16, 0.054 m06b483cf: five against three Euler steps, 0.154 m09ad74a209ad74a2: spread among the six FP8 candidates, mean 0.98 m, max 2.63 m09ad74a2: FP8 against NVFP4 W4A16, 0.033 m09ad74a2: five against three Euler steps, 0.052 m0a1ef8080a1ef808: spread among the six FP8 candidates, mean 0.70 m, max 1.70 m0a1ef808: FP8 against NVFP4 W4A16, 0.073 m0a1ef808: five against three Euler steps, 0.078 m0ea6fd880ea6fd88: spread among the six FP8 candidates, mean 0.94 m, max 2.49 m0ea6fd88: FP8 against NVFP4 W4A16, 0.074 m0ea6fd88: five against three Euler steps, 0.182 m174407b0174407b0: spread among the six FP8 candidates, mean 0.91 m, max 2.25 m174407b0: FP8 against NVFP4 W4A16, 0.120 m174407b0: five against three Euler steps, 0.083 m195f1fc6195f1fc6: spread among the six FP8 candidates, mean 1.30 m, max 2.95 m195f1fc6: FP8 against NVFP4 W4A16, 0.205 m195f1fc6: five against three Euler steps, 0.586 mea7bbd31ea7bbd31: spread among the six FP8 candidates, mean 0.75 m, max 1.83 mea7bbd31: FP8 against NVFP4 W4A16, 0.028 mea7bbd31: five against three Euler steps, 0.051 m
clipFP8 vs NVFP4, m5 vs 3 steps, m6 vs 1 candidate, mcandidate spread, mean / max, mFP8 vs PyTorch BF16, m
030c760c, following a slowing lead vehicle0.0080.0350.0040.76 / 1.950.18
0347d9f9, right turn0.3290.1870.0091.66 / 3.581.52
06b483cf, straight through an intersection0.0540.1540.0031.55 / 4.140.49
09ad74a2, straight, green light0.0330.0520.0050.98 / 2.630.77
0a1ef8080.0730.0780.0050.70 / 1.700.78
0ea6fd880.0740.1820.0020.94 / 2.490.41
174407b0, all-way stop0.1200.0830.0040.91 / 2.250.88
195f1fc6, left turn0.2050.5860.0111.30 / 2.950.97
ea7bbd31, NVIDIA's example, right turn0.0280.0510.0040.75 / 1.83–

The PyTorch column compares against case study 06's BF16 waypoints for the same clip, t0 and navigation text, but with a different noise draw (seed 43 against seed 0) and one sample, so it is a sanity bound rather than a quantization error. A like-for-like reference needs the same noise tensor fed to both runtimes.

What each change bought

Energy per decision

Board input rail by tegrastats at 100 ms, integrated over the timed block and divided by the decisions; the mean power while deciding is printed above each column.

measured herepublished, not run here
Energy per decision, by build02040joules per decisionFP8 recipe as written: 29.1 J per decision, 85 W mean29.1 J85 W meanFP8 recipe aswrittenFP8, three Euler steps: 29.4 J per decision, 86 W mean29.4 J86 W meanFP8, threeEuler stepsFP8, six candidates: 33.4 J per decision, 84 W mean33.4 J84 W meanFP8, sixcandidatesNVFP4 W4A16 language model: 37.2 J per decision, 84 W mean37.2 J84 W meanNVFP4 W4A16language modelNVFP4 W4A16, six candidates: 42.1 J per decision, 85 W mean42.1 J85 W meanNVFP4 W4A16,six candidatesEmbedl card, same board: 12.4 J per decision, published12.4 JpublishedEmbedl card,same board

Issues found, with causes

In the order they happened. The full log with times is FRICTION_LOG.md.

What stands between this build and the published 92 ms

Provenance

Modelnvidia/Alpamayo2-Edge, revision aaaa32c6 (the recipe's pin; both weight files equal by sha256 to the 2026-11-10 copy)
RecipeNVlabs/alpamayo-recipes 622f9d2, recipes/alpamayo2edge_edgellm; TensorRT Edge-LLM v0.11.1 (358de11c), plugin rebuilt with -DENABLE_CUTE_DSL=ALL and nvcc 13.0
BoardJetson AGX Thor (label bench-thor), JetPack 7.1-b112, kernel 6.8.12-tegra, CUDA 13.0, TensorRT 10.13.3.9, torch 2.10.0+cu130, NV Power Mode: MAXN, CPU governor schedutil, GPU 1575 MHz throughout
Engine buildsFP8: Edge-LLM compile 80 s, language engine 46 s, vision engine 32 s, K/V capacity 3584 tokens. NVFP4 W4A16: language engine 43 s, vision 41 s
CalibrationFP8 and NVFP4 scales on the 22 frames of NVIDIA's example clip, as the recipe does; FP8 K/V cache; reduced 8192-row output vocabulary
Timingtools/thor_stream.py: engines loaded once, decisions back to back, one untimed warm-up per clip, stage times by stream synchronization; power by tegrastats at 100 ms on the board input rail, energy integrated over the timed block
DataPhysicalAI-Autonomous-Vehicles (gated, NVIDIA AV dataset licence): clip ids and derived navigation text only; no frames leave the workstation