EmbodiedEdge Evidence
Runtime safety · Robot workcells · Jetson AGX Thor · The full build Functional · live-camera validation in progress

Physical AI Safety Observability

The full build

Cameras watch a robot workcell. A vision-language model on the device reports who is in the frame and what they are wearing, safety rules turn that into structured events, and an operator reviews them. The full build adds the ways video gets in, an access gate that checks a worker's protective gear, a labeled scorer, and Cosmos 3 runs. It measures what it costs to run, and it records what the model gets wrong. The source of the full build is not public; the open core is.

DeviceJetson AGX ThorL4T R38.4.0, 120 W mode
ModelsCosmos-Reason2 2B and 8B, Cosmos 3 Edgeserved on device by vLLM
Rules5 safety rulesPPE, zone, proximity, path, summary
EvidenceA committed record for every numberunder reports/thor/
Watch it run

Protective gear checked on the Jetson AGX Thor

The video shows the access gate reading a worker's protective gear with Cosmos Reason 2 on the Thor.

What the full build adds

More ways in, a gate, and a way to score the model

Any video in

Paste an RTSP link with its credentials taken out and stored encrypted, pick a USB camera with its sizes and frame rates listed, upload an MP4, MOV or MKV, or share a phone or laptop camera from the browser. All of them run through the same rules and events.

Every source labelled

Each camera, event and run report records where its video came from. Events and evidence frames from uploaded or synthetic sources carry a visible "not a live camera" note on Live, Events and in the run report.

An access gate, phase 1

A front-scan check of hard hat, glasses, vest and boots, read region by region with two question wordings that must agree. Camera loss, staleness, tailgating, departure, timeout, uncertainty and disagreement all deny. It is a compliance aid, not a safety interlock, and its output is simulated and closed by default.

A second reasoner

Cosmos 3 Edge is a selectable backend beside Cosmos Reason 2, which stays the default. Both receive the same prompt, schema and settings.

A labeled scorer

Scores the rules against 60 labeled simulation pictures, with every request and answer kept as evidence.

Measured runs from the console

The Runs page records a run in the same format as the committed Thor runs. A run refuses to start when the model server is not ready and stops, marked failed, if the server goes down.

Cell plan

What the rules look for in a workcell

The operator draws restricted zones on a snapshot from each camera and picks which rules run on it. Every rule works on what the model reports for that frame: labels, boxes, and for each person whether a hard hat and a high-visibility vest are worn.

camera robot hat ✗ vest ✓ hat ✓ vest ✓ EXIT pallet 1 PPE MISSING 2 RESTRICTED ZONE ENTRY 3 ROBOT PROXIMITY 4 BLOCKED EMERGENCY PATH Schematic. Not from a run.
The worker in the cell trips three rules at onceProximity is measured in camera pixels
  • PPE missing HIGH

    A person without a hard hat or without a high-visibility vest. The blue circle is the ISO 7010 shape for a mandatory action.

  • Restricted zone entry CRITICAL

    A person's box overlaps a zone the operator drew as restricted, such as the robot cell.

  • Human and robot proximity HIGH

    The centre of a person's box within 90 pixels of the centre of a robot's box.

  • Blocked emergency path MEDIUM

    A pallet, cart or box the model reports as blocking an emergency path.

  • Unsafe event summary LOW

    Anything else the model itself calls unsafe, passed through for review.

Signal path, per camera

Three loops that never wait on each other

CaptureNative frame rate

One thread per camera decodes the stream and keeps the newest frame. The console shows that video at the camera's own rate.

InferenceOn its own interval

A worker samples the newest frame, asks the model for people, objects and PPE, and runs the rules bound to that camera.

PostingAsynchronous

Events go to a bounded queue drained by its own thread, with retries, a drop counter, and latency measured from frame capture to the backend's acknowledgement.

A model outage never invents an event

If the model server stops, the video keeps playing and the console shows "inference unavailable". Events only come from frames the model actually answered.

Logic as shipped

When a person is flagged, and when a human must look

Read from rules/policies.py and rules/severity.py. The rule functions are pure: no I/O, the same detections always give the same findings.

PPE check

for each person: if not hard_hat or not vest: PPE_MISSING, HIGH confidence = highest among those people

Hard hat and vest only. Gloves, glasses and boots are not checked.

Review gate

review required if severity is HIGH or CRITICAL or confidence < 0.85 or runtime is not nominal

A degraded runtime lowers confidence by 0.08, a critical one by 0.18, and each dropped frame by 0.01 up to 0.10.

Measured on Jetson AGX Thor, 2026-09-09

The pipeline keeps camera rate. The model sets the pace.

Every number below is read from a committed run artifact. The model runs used 60 frames of an Isaac Sim workcell recording with a robot arm and no people in it.

28.588 fpsPipeline at 30 fps pacing with the mock model, 1,800 framesmock_30fps_no_post.json
0.9877 msRule evaluation per frame, p95mock_30fps_no_post.json
2.69 sCosmos-Reason2-2B per frame, p50, with the JSON grammarcosmos2b_video_60_schema.json
62.5 WBoard power p50 during that run, against about 24 W idlecosmos2b_video_60_schema_tegrastats.jsonl

Inference latency per frame, five model runs

p50p95
0 s51015202530 s 2B, think block2B, no think2B, JSON grammar8B, JSON grammar8B, no think 2B, think block · p50 4.35 s2B, think block · p95 12.85 s 2B, no think · p50 3.57 s2B, no think · p95 6.58 s 2B, JSON grammar · p50 2.69 s2B, JSON grammar · p95 3.36 s 8B, JSON grammar · p50 3.20 s8B, JSON grammar · p95 8.26 s 8B, no think · p50 8.29 s8B, no think · p95 27.78 s 12.856.583.368.2627.78

Hover a dot for its value. A sixth run is left off on purpose: a grammar with a mismatched prompt made the model return nothing on every frame, so its 0.31 s p50 is the cost of generating nothing. The README keeps it as a labelled failure case.

The grammar fixed the labels and exposed phantom people

With a JSON-schema grammar, out-of-vocabulary labels went from 59 to 0 on the 2B. On footage with no people, it then reported six persons and fired seven false PPE and proximity events.

The 8B did not see anyone who was not there

Under the same grammar and prompt, the 8B reported zero persons and zero events on the same frames, at 3.20 s p50. Sixty frames of one video is a signal, not a rate.

Measured on the Thor, 2026-10-02

The config refuses a schema the server would ignore

With vLLM 0.14's reasoning parser on, a JSON schema is accepted but not enforced. That is now blocked in code: before any worker uses constrained mode, a canary request asks for one word while the schema allows only another, and the server has to answer with the schema's value. The probe is committed as reports/thor/constrained_guard_probe.json.

HTTP 422

Constrained mode refused when starting the reasoning-parser container.

HTTP 422

Constrained mode refused when saving settings with the reasoning parser on.

"banana"

The reasoning-parser server's answer to the canary. The guard refused.

schema value

The container without the parser honoured the grammar and was allowed. Ready 55.6 s after launch.

Labeled simulation pictures, 2026-10-17 and 2026-11-14

Scored against known answers, then run through Cosmos 3

60 Isaac Sim pictures in four conditions: a worker with a hard hat and vest, a worker with neither, a worker inside a restricted zone, and an empty scene. Measured on the Thor with the full build's labeled scorer; the summary reports are published in the open core under reports/thor/. Simulation, not real-world footage.

11 → 0False protective-gear alarms after hard_hat and vest became required fields. All 15 real ones keptmodel_accuracy_ppe_v1_2026-10-17.json
29 → 44People found, of 45: Cosmos Reason 2 (2B) against Cosmos 3 Edge (4B), same requestreasoner_comparison_2026-11-14-compact-grammar.json
16 → 0False "too close to the robot" alarms, Reason 2 against Cosmos 3 Edgereasoner_comparison_2026-11-14-compact-grammar.json
39 of 60Cosmos 3 Edge answers with nothing usable when launched as the model card writes it. One server setting took it to 0reasoner_comparison_2026-11-14.json
Cosmos 3 Edge found the people Reason 2 missed

It found the worker in 44 of 45 pictures against 29, found 14 of 15 zone entries against 9, and raised none of the false proximity alarms. Median answer time 1.83 s against 1.54 s.

It read protective gear worse

Cosmos 3 Edge kept 13 of 15 real gear alarms and raised 3 false ones, where Reason 2 kept 15 and raised none on the people it did find. It also needed disable_any_whitespace in the server's grammar settings to answer at all.

Darker scenes, redrawn with Cosmos Transfer

Cosmos3-Nano-Sim-Transfer redrew the full build's access-gate test clips as daylight, dusk and night on the Thor. The gate's decision held at 15 of 15 in every version. The safety-glasses read fell from 45 to 27 of 45 by night.

Cosmos Transfer on the Thor, 2026-11-15

What the gate reads when the scene gets darker

The gate's reader scored 15 of 15 on five daylight test clips. Cosmos3-Nano-Sim-Transfer redrew the same clips as daylight, dusk and night on the Thor, keeping each worker's position and gear, and the gate's own check scored every version three times. Generated test clips and model redraws of them, not field evidence.

Original. A worker with no hard hat.
Dusk. Redrawn by Cosmos Transfer.
Night prompt. A dim evening, about half the original brightness.
VersionMean brightness (0 to 255)Open or denied right (of 15)Let through without gearSafety-glasses reads right (of 45)
Original, resized to 832 x 480141 to 14515, 15, 15045
Daylight redraw132 to 14015, 15, 15039
Dusk84 to 9115, 15, 15038
Night54 to 7215, 15, 15027
The decision held

With Cosmos Reason 2 as the reader, open or denied was right on every sample in every lighting, and no worker without gear was let through. Hard hat, vest and boots were read correctly throughout.

Clear glasses went first

The safety glasses, the smallest and faintest item, lost 18 of 45 reads by night. Part of that is the redraw itself: the daylight redraw already lost 6. Glasses only raise a warning at this gate.

Cosmos 3 Edge is not a drop-in reader here

Behind the same gate with nothing else changed, it denied every compliant worker in every version, the untouched originals included. The gate's question wordings were written and checked against Cosmos Reason 2.

Five gate clips: original, daylight redraw, dusk and night.
All five clips. Columns: original, daylight redraw, dusk, night. Rows: full gear with glasses, without glasses, no hard hat, no vest, trainers. Each clip took 148 to 156 s and 3.5 to 3.8 Wh to generate.
Measured on the Thor, 2026-10-30, under a frozen protocol

Three passes, the same result, and the frame that stopped them

The constrained Cosmos Reason 2 configuration repeated as three passes behind a warm-up, with the server image, model files, source commit and the board's other load recorded before each run.

2.33 sAnswer time per frame, p50, median of three passes (p95 3.30 s)cosmos_cost_v1_2026-10-30.json
22.0Frames per minutecosmos_cost_v1_2026-10-30.json
60.1 WMean board input powercosmos_cost_v1_2026-10-30.json
164 JEnergy per framecosmos_cost_v1_2026-10-30.json
Every pass stopped at the 45th frame

The server returned 512 tokens cut off mid-string, and the worker fails a run on the first answer that breaks the output contract, by design. The 44 frames before it are the cost measurement; the 45th is the finding.

Operator console

Watch, filter and review in one place

Screens from the running app, captured with its built-in synthetic workcell feed and the mock detector. The console labels both on screen so they are never mistaken for a real camera or a real model.

Live page: synthetic workcell feed with boxes for a person marked hat missing and vest present, a robot and a pallet, a dashed restricted zone, a red No PPE banner and a column of high severity events marked review required.
Live. Each person's box shows what the model reported for the hard hat and the vest. Events land in the side column as they post, marked when a human must review them. The strip below shows frames, drops, the model and the poster's queue.
Events page: a table of safety events with time, severity, rule, camera, confidence and review status, with filters for camera, rule, severity and review required.
Events. Filter by camera, rule, severity or review required. Repeated events are grouped on the Incidents tab, and each event keeps the frame the model saw.
Model page: host memory and Docker status, the model server state, and a catalog of three Cosmos-Reason2 containers, one marked reasoning parser with no constrained mode.
Model. The catalog checks memory before anything starts and marks the reasoning-parser container as think mode only. Captured on a machine without the weights, so each shows weights missing.
Built for a real deployment

Details that matter on a factory floor

Every number has a file

Each run writes one artifact with the device, date, inputs, per-frame timings, token use, event counts, peak memory and a 1 Hz tegrastats record.

Failures are kept

A timed-out run, an empty-output run and a model echoing its own schema are all committed and labelled, not edited out.

Cameras set up in the UI

Ten vendor profiles, a connection test that decodes one frame or names the problem, and zones drawn on a live snapshot.

Credentials stay put

Passwords are encrypted at rest, never returned by the API, only reused for the host they were saved for, and masked in logs, events and reports.

Shared model server guarded

The model container keeps running across restarts. Stopping it needs an explicit, logged confirmation.

Nothing acts on its own

The system flags and records. It never stops a robot or a line; a person decides.

Status, November 2026

What is proven, and what is next

Working now
  • Runs end to end from camera setup to events in the console, and is deployed on the Thor as a service.
  • Runtime cost measured: pipeline overhead, model latency, power and temperature for 2B and 8B.
  • Constrained-mode guard measured against the real containers.
  • Scored on labeled simulation pictures with Cosmos Reason 2 and Cosmos 3 Edge, in the full build.
  • Access gate, phase 1, on uploaded test clips: 15 of 15 decisions right in daylight, and the decision held in every darker redraw.
  • Every video source: vendor profiles, RTSP links, USB, uploads and browser cameras, each labelled.
  • Tests and CI on every push to main.
Not yet measured
  • No live-camera run is committed. Capture rate, dropped frames, packet-to-event latency and power with a camera attached are unknown.
  • No rule accuracy on real footage. Rules are rated only on 60 labeled simulation pictures.
  • The asynchronous poster has not been measured on the Thor.
  • Long-run stability. Committed runs last 60 to 390 seconds.
Planned upgrades
Live-camera evidence run on the Thor PPE check against known poses: vest, hat, both, neither Cosmos-Reason2-8B on live video Zones from camera calibration

Out of scope by design: stopping machines or acting without a person. This is observability with a human in the loop.

An EmbodiedEdge project. Measured on real hardware, published as found.