Bench2Field · Case study 07
Alpamayo 2 Edge with TensorRT Edge-LLM on the Jetson AGX Thor
Where every build lands
Per decision, back to back with the engines loaded once, five Euler steps, the six newest camera frames encoded per decision (the recipe's rolling mode), MAXN. Nine clips, 20 decisions each: the eight of case studies 02 and 06 and NVIDIA's example clip. The PyTorch row is case study 06's 10 Hz stream on the same board (30 decisions). Hatched rows are published figures, not measured here.
| build | median, ms | p95, ms | J per decision | mean W | record |
|---|---|---|---|---|---|
| PyTorch stream, 10 Nov (case study 06) | 331 | 544 | – | – | 06_alpamayo2_edge/runs/stream_bench-thor_tuned_fp8_steps5_maxn.json |
| FP8 recipe as written | 367 | 427 | 29.1 | 85 | runs/stream/clips9_rolling_c1.json |
| FP8, three Euler steps | 353 | 512 | 29.4 | 86 | runs/stream/clips9_rolling_c1_s3.json |
| FP8, six candidates | 413 | 493 | 33.4 | 84 | runs/stream/clips9_rolling_c6.json |
| NVFP4 W4A16 language model | 462 | 593 | 37.2 | 84 | runs/stream/clips9_rolling_c1_nvfp4-w4a16.json |
| NVFP4 W4A16, six candidates | 512 | 627 | 42.1 | 85 | runs/stream/clips9_rolling_c6_nvfp4-w4a16.json |
| NVIDIA, DRIVE AGX Thor, with drafter | 220 | – | – | – | published: indicative, recipe README; range 180 to 250 ms over ten clips |
| Embedl card, same board | 92 | 140 (p90) | 12.4 | – | published: model card, 2026-11-11: NVFP4 language model, MXFP8 vision and expert, caches, speculation; one clip, 180 decisions |
Where a decision's time goes
Stage medians, stream-synchronized at each boundary as the recipe's own runner does. The chain of causation is greedy decoding, one token at a time; the recipe's speculative drafter is not released for the Edge model.
With no reasoning at all the floor is vision, prefill and expert: about 155 ms at one candidate, already above Embedl's published median. Each token of reasoning costs 10.8 ms of GPU work (measured with the engine alone, with and without a host sync per token), so a 22-token sentence is about 237 ms of the decision. The NVFP4 engine holds 1.62 GB of packed weights against 1.98 GB in FP8, yet its token costs 15.0 ms on the example clip against 11.0 in FP8: on Edge-LLM 0.11.1 with TensorRT 10.13.3.9 the single-token NVFP4-A16 kernel does not turn halved bytes into time, and W4A16 adds dequantization work to the compute-bound prefill.
The nine clips
FP8 recipe, one candidate, median of 20 decisions per clip. Four clips take the model's no-reasoning path (<NOMINAL>, 6 tokens) and finish in about 220 ms; the five that write a sentence take 367 to 427 ms.
| clip | median, ms | tokens | chain of causation (FP8; NVFP4 identical unless noted) |
|---|---|---|---|
| 030c760c, following a slowing lead vehicle | 427 | 26 | Decelerate to maintain a safe following distance behind the slowing lead vehicle and prepare to stop for the red traffic light ahead. NVFP4 differs: "Decelerate to maintain a safe following distance behind the lead vehicle as it slows for the cyclist and traffic queue ahead." |
| 06b483cf, straight through an intersection | 395 | 22 | Proceed straight through the intersection to follow our route, as the roadway and crosswalk are clear. |
| ea7bbd31, NVIDIA's example, right turn | 394 | 22 | Decelerate and turn right at the intersection to follow our route, as the roadway is clear. NVFP4 differs: "Decelerate and turn right at the intersection to follow our route, as the roadway is clear of conflicting traffic." |
| 174407b0, all-way stop | 381 | 21 | Decelerate to stop at the stop line for the all-way stop sign at the upcoming intersection. |
| 09ad74a2, straight, green light | 367 | 19 | Proceed straight through the intersection to follow our route, as the traffic signal is green. NVFP4 differs: "Proceed straight through the intersection to follow our route, as the traffic light is green." |
| 0ea6fd88 | 222 | 6 | <NOMINAL> |
| 0a1ef808 | 221 | 6 | <NOMINAL> |
| 195f1fc6, left turn | 220 | 6 | <NOMINAL> |
| 0347d9f9, right turn | 218 | 6 | <NOMINAL> |
Did the faster builds still drive the same way?
Trajectories decoded with the recipe's host/compare.py from the action expert's output, ADE in metres over the 64 waypoints, same noise seed across builds. The grey bar is the spread among the six candidates the FP8 engines sample for the same clip (mean marked), the study's budget: a build whose difference sits inside it changes the plan less than the model's own sampling does.
| clip | FP8 vs NVFP4, m | 5 vs 3 steps, m | 6 vs 1 candidate, m | candidate spread, mean / max, m | FP8 vs PyTorch BF16, m |
|---|---|---|---|---|---|
| 030c760c, following a slowing lead vehicle | 0.008 | 0.035 | 0.004 | 0.76 / 1.95 | 0.18 |
| 0347d9f9, right turn | 0.329 | 0.187 | 0.009 | 1.66 / 3.58 | 1.52 |
| 06b483cf, straight through an intersection | 0.054 | 0.154 | 0.003 | 1.55 / 4.14 | 0.49 |
| 09ad74a2, straight, green light | 0.033 | 0.052 | 0.005 | 0.98 / 2.63 | 0.77 |
| 0a1ef808 | 0.073 | 0.078 | 0.005 | 0.70 / 1.70 | 0.78 |
| 0ea6fd88 | 0.074 | 0.182 | 0.002 | 0.94 / 2.49 | 0.41 |
| 174407b0, all-way stop | 0.120 | 0.083 | 0.004 | 0.91 / 2.25 | 0.88 |
| 195f1fc6, left turn | 0.205 | 0.586 | 0.011 | 1.30 / 2.95 | 0.97 |
| ea7bbd31, NVIDIA's example, right turn | 0.028 | 0.051 | 0.004 | 0.75 / 1.83 | – |
The PyTorch column compares against case study 06's BF16 waypoints for the same clip, t0 and navigation text, but with a different noise draw (seed 43 against seed 0) and one sample, so it is a sanity bound rather than a quantization error. A like-for-like reference needs the same noise tensor fed to both runtimes.
What each change bought
- NVIDIA's recipe on the Jetson boardTail 544 to 427 ms against the PyTorch stream; median 331 to 367 ms, so no gain at the median.
- Vision cache across decisionsVision stage 98 to 27 ms; p95 613 to 525 ms and 8 J less per decision against encoding all 22 frames; the median moved only 9 ms because the cold run decoded faster on that clip (example clip, six candidates).
- One candidate instead of sixExpert 83 to 34 ms; decision 413 to 367 ms median.
- Three Euler steps instead of fiveExpert 34 to 25 ms; median 367 to 353 ms; trajectories within 0.2 m of five steps on eight clips, 0.6 m on one.
- NVFP4 W4A16 language modelDecode 11.0 to 15.0 ms per token on the example clip, prefill 95 to 124 ms, vision 25 to 33 ms. The engine file is smaller (1.62 against 1.98 GB), so the weights are packed; the single-token kernel on this stack does not turn the bytes into time.
- Token kept on the GPU, end of chain checked per chunk286 ms decode against 242: no change. The host round trip was never the cost.
Energy per decision
Board input rail by tegrastats at 100 ms, integrated over the timed block and divided by the decisions; the mean power while deciding is printed above each column.
Issues found, with causes
In the order they happened. The full log with times is FRICTION_LOG.md.
- 1. The recipe's submodule step must be recursivecmake stopped on a missing xgrammar header: Edge-LLM's own submodules were absent.Fetched recursively, copied 3rdParty to the board.
- 2. nvcc from the `cuda` alternative against CUDA 13.0 headersThe toolchain forces /usr/local/cuda/bin/nvcc, 13.2 on this board; its launch stubs do not match 13.0's host_runtime.h, every CUDA file failed.nvcc pinned to the 13.0 toolkit, CUDA_DIR and cuRAND include set to match.
- 3. Action engine fixed at six candidatesA one-candidate export writes candidates_per_sequence 2; the builder computes zero sequences and TensorRT rejects the profile.Set candidates_per_sequence to 1 in the one-candidate config.
- 4. Shim renames MLP layers, copies the quantization config verbatimA MIXED_PRECISION (NVFP4) config loses every per-layer entry; the exporter treats packed weights as FP16 and torch.export fails on a shape mismatch.Rewrote the config keys with the same renames, dropped modules restored to FP16.
- 5. NVFP4-A16 GEMM kernels need the CuTe DSL build flagNvfp4A16BlackwellGemmRunner::isSupported is false unless the plugin is built with ENABLE_CUTE_DSL=ALL; the recipe's build script leaves the default.Reconfigured with the kernel groups on; the W4A16 engines built.
- 6. TensorRT 10.13 cannot build the W4A4 graphMyelin stops on PLUGIN_V3 with empty producers; the recipe pins TensorRT 11.3.Parked; a newer TensorRT (JetPack 7.2.2) is the owner's call.
- 7. A probe reported 1.1 ms per decode tokenA timing bug in the probe (start time reset per iteration). The corrected figure is 11.8 ms, and the wrong number was withdrawn the same hour.Probe fixed; both numbers kept in the friction log.
What stands between this build and the published 92 ms
- A speculative drafterNVIDIA's 220 ms uses a DFlash2 drafter that is not part of the model release. Greedy decoding costs 10.8 ms per token here; a drafter verifies several tokens per weight read. Training one with NVIDIA's SpecForge recipe is a project of its own.
- A working FP4 decode pathW4A16 is slower on this stack and W4A4 does not build on TensorRT 10.13.3.9. A newer TensorRT (JetPack 7.2.2) is the next thing to try, and a system change for the owner to decide.
- A shorter prefill95 ms of every decision re-encodes a prompt in which old and new camera frames interleave per camera, so unchanged frames are not a cacheable prefix. A prompt layout that keeps them contiguous, or Edge-LLM's media-aware K/V reuse, is the lever.
Provenance
| Model | nvidia/Alpamayo2-Edge, revision aaaa32c6 (the recipe's pin; both weight files equal by sha256 to the 2026-11-10 copy) |
|---|---|
| Recipe | NVlabs/alpamayo-recipes 622f9d2, recipes/alpamayo2edge_edgellm; TensorRT Edge-LLM v0.11.1 (358de11c), plugin rebuilt with -DENABLE_CUTE_DSL=ALL and nvcc 13.0 |
| Board | Jetson AGX Thor (label bench-thor), JetPack 7.1-b112, kernel 6.8.12-tegra, CUDA 13.0, TensorRT 10.13.3.9, torch 2.10.0+cu130, NV Power Mode: MAXN, CPU governor schedutil, GPU 1575 MHz throughout |
| Engine builds | FP8: Edge-LLM compile 80 s, language engine 46 s, vision engine 32 s, K/V capacity 3584 tokens. NVFP4 W4A16: language engine 43 s, vision 41 s |
| Calibration | FP8 and NVFP4 scales on the 22 frames of NVIDIA's example clip, as the recipe does; FP8 K/V cache; reduced 8192-row output vocabulary |
| Timing | tools/thor_stream.py: engines loaded once, decisions back to back, one untimed warm-up per clip, stage times by stream synchronization; power by tegrastats at 100 ms on the board input rail, energy integrated over the timed block |
| Data | PhysicalAI-Autonomous-Vehicles (gated, NVIDIA AV dataset licence): clip ids and derived navigation text only; no frames leave the workstation |