EmbodiedEdge Evidence
Edge vision analytics · Streets and intersections · The full build Functional · field validation in progress

Urban Edge Vision

The full build

Street cameras are set up in a web console. A vision-language model on the device looks at each frame, use-case packs turn what it sees into traffic events, and an operator reviews them. The full build adds the ways video gets in, ground-plane tracking, cloud delivery that survives outages, and Cosmos 3 runs that generate test events. Processing stays on the device, and nothing is enforced automatically. The source of the full build is not public; the open core is.

Field deviceJetson AGX ThorCosmos-Reason2-2B on vLLM; Cosmos 3 Edge for test generation
WorkstationRTX 5090 class GPUGemma 4 on Ollama or vLLM
Video in10 vendor profilesplus RTSP links, USB, uploads, phone cameras
Packs4 use casesper camera, combinable
Watch it run

The console, live on the Jetson AGX Thor

The video walks through the system on the Thor. The clips below are cut from screen recordings of the console on 2026-11-04. Each one plays an uploaded generated clip through a local CPU demo detector, and the console labels every frame "Uploaded · Generated, not a live camera".

Stop sign, full stop. A car is tracked through the drawn stop zone and called a full stop from its dwell time and slowest speed.
Stop sign, no stop. The same pack on a car that does not stop; the event goes to the Review queue.
Moving object. Pedestrians tracked with direction and a short descriptor, listed as they post.
Vehicle count. Each tracked vehicle counted once as it crosses the line, by direction and class.
What the full build adds

More ways in, motion on the road, delivery that survives outages

Any video in

Paste an RTSP link with its credentials taken out and stored encrypted, pick a USB camera with its sizes and frame rates listed, upload an MP4, MOV or MKV, or share a phone or laptop camera from the browser. All of them run through the same packs and review.

Every source labelled

Each camera, event and run records where its video came from: live RTSP, USB, browser, uploaded recording, uploaded generated footage or synthetic. Anything that is not a live camera carries a visible label on Live, Events, Review and in the run record.

Positions on the road

An optional per-camera add-on projects vehicles onto a calibrated flat road and estimates position, trajectory, speed in m/s and mph, heading, and a one to two second motion prediction. The packs and their events keep their own behaviour.

Pixel motion, measured

Optional optical flow measures how image features move inside each vehicle box and checks it forward and backward. Weak texture or a gap gives "unknown", never a zero speed.

Delivery that survives outages

Messages wait in a bounded outbox on disk until the broker acknowledges them, with retries across restarts. When the quota runs out the newest messages are rejected visibly, not silently.

Gaps shown, not hidden

Old or unprojectable measurements are withheld, speed warm-up restarts after a gap, and Live shows motion freshness, usable speed coverage, cloud backlog and delivered messages.

Site plan

Four packs, drawn on the street

Each pack works on geometry the operator draws in the Studio page over a live snapshot from the camera. Coordinates are stored normalized to the frame, so they survive a change of stream resolution.

STOP t12 1 STOP ZONE A B distance entered in metres t31 → 2 SPEED BETWEEN GATES A B Eastbound Westbound ← t44 bus t45 3 VEHICLE COUNT p7 → 4 MOVING OBJECT ZONE N ↑ Schematic. Track ids are illustrative, not from a run.
Geometry stored 0 to 1, relative to the frameOne camera can carry several packs
  1. 1
    Stop sign

    Watches each vehicle track through the stop zone and calls it a full stop, a rolling stop or no stop from its dwell time and slowest speed in the zone.

  2. 2
    Speed between two gates

    Times a track from gate A to gate B and converts the distance the operator entered into km/h or mph. Only measured crossings produce a reading.

  3. 3
    Vehicle count

    Counts each tracked vehicle once as it crosses the A to B line, by direction and by class: car, truck, bus, motorcycle. Hourly totals per camera.

  4. 4
    Moving object

    Tracks people or other chosen classes with direction and a short descriptor, across the whole frame or inside a drawn zone.

Speed and stop sign are refused on the same camera. They need different sight lines, so the console blocks the combination instead of producing weak results for both.

Signal path, per camera

From stream to reviewed event

01 · captureDecode at native rate

One capture thread per camera writes the newest frame to a single slot.

02 · watchLive video

The browser reads that slot as MJPEG at the camera's own frame rate.

03 · lookModel on its own cadence

Inference samples the slot every interval. vLLM, Ollama or NVIDIA NIM, switchable live.

04 · decideTracker and packs

Detections get stable track ids, then each pack bound to the camera applies its rule.

05 · reviewOperator queue

Events persist in SQLite. Flagged ones wait in Review with the frame the model saw.

Video never waits on the model

The model server is not in the video path. If it stops or falls behind, the Live page keeps playing and shows "inference unavailable" instead of inventing events. An end-to-end runtime test checks exactly that.

Decision rules, as shipped

What each pack actually decides

The defaults below are read from the pack code. Every one can be changed per camera in Studio.

Stop sign

full stop slowest ≤ 0.02 frame widths/s and dwell ≥ 1,000 ms rolling stop slowest ≤ 0.08 no stop anything faster
  • Decided once, when the track leaves the zone
  • Rolling and no stop go to the Review queue

Speed between two gates

km/h = distance_m ÷ (t_B − t_A) × 3.6 discard if transit > 30 s compare with posted limit (default 50 km/h)
  • Distance comes from a tape measure or a map, entered once
  • Readings shown in km/h or mph

Vehicle count

count once per crossing of A→B or B→A track seen ≥ 2 times before counting jitter on the line = one record
  • Direction labels are the operator's, such as Eastbound
  • Counts never enter the Review queue

Moving object

classes pedestrian by default zone whole frame, or a drawn polygon report direction, speed band, descriptor
  • Useful near crosswalks and sidewalks
  • Works on any camera, street view or not
Operator console

Set up, watch and review in one place

Screens from the running app. These captures use the built-in synthetic test feed and the mock detector, and the console labels both on screen so they are never mistaken for a real camera.

Live page: synthetic camera with detections, the dashed count line from A to B, vehicle counts by direction and a feed of recent events.
Live. Video at the camera's frame rate with detections and the count line drawn over it. Vehicle counts by class and direction, recent events, and per-camera health: frame rate, dropped frames, reconnects, packs running.
Add camera form with the Hikvision profile selected, a documentation-range host address, port 554 and the stream path filled in from the profile.
Cameras. Pick a vendor profile and the stream path fills itself in for main or sub stream. Test connection decodes one frame or names the problem: unreachable, wrong password, wrong path, codec.
Use-case Studio with Moving Object, Stop Sign and Vehicle Count selected and Speed Violation marked blocked.
Studio. Choose packs per camera and draw their zones, gates and count line on a snapshot. Here Speed is blocked because Stop sign is selected on the same camera.
Models page showing the host's GPU and memory, the current model's health, and a catalog where models that will not fit are marked with the reason.
Models. The catalog is checked against the host before anything loads: free VRAM on a discrete GPU, available memory on Jetson. Models that will not fit are shown blocked with the reason. On Jetson the console can launch NVIDIA's vLLM container itself.
Built for a real deployment

Details that matter once it is on a pole

Credentials stay put

Camera passwords are encrypted at rest with a key file outside the repository. The API never returns one, and a saved password is only reused for the host and port it was saved for.

Nothing leaks into logs

Stream URLs are masked in logs, errors, status payloads and run artifacts.

Right model for the box

On first start the console picks a default for the hardware: Cosmos-Reason2-2B on Thor, Gemma 4 on a workstation. The operator can switch at any time.

Shared model server guarded

A model container another app depends on cannot be stopped by accident. Stopping it needs an explicit, logged confirmation.

Runs as a service

A systemd unit starts the API and console at boot and keeps a persistent log.

People decide

Packs flag, operators confirm or dismiss. Each item can carry a ground-truth note, such as "my car, full stop", so pack output can be checked against known passes.

Cosmos 3 on the Jetson AGX Thor, 2026-11-14 and 2026-11-15

Rare events on demand, and a prediction test

Rare, dangerous events cannot be filmed on purpose. Cosmos 3 Edge (4B) draws them from a still on the Thor, and the production packs score the result. Generated footage throughout: offline test data, not live prediction, and no real-world accuracy.

EventPrompt as written, then upsampledHand-written promptHand-written prompt and a reference frame
Pedestrian steps out from behind a parked pickupPartlyYes
Car rolls through the stopPartlyYes
Car runs the stop with cross trafficNoNoYes

Checked by a person watching each clip. The framework's prompt upsampler had turned "never stops" into "stops precisely at the STOP line". Hand-written prompts with compass directions, exclusions and timing fixed the single-actor events. The cross-traffic event also needed the second vehicle placed in the starting frame, because an actor that exists only in words is placed wherever the model chooses. Each clip took 114 to 125 s on the board at 2.6 to 2.8 Wh.

The starting frame as it was, and with a white van placed on the west side of the cross street.
The reference frame. Left: the original still. Right: the same still with the van placed on the cross street, the first stage of a two-stage generation.
Five frames of the round-two cross-traffic clip.
Words only. The van appears beside the camera and crosses the near crosswalk; the sedan never moves.
Five frames of the round-three cross-traffic clip.
With the reference frame. The van stays on the cross street and crosses west to east in front of the sedan.
Predicting the next four seconds

Given only the first second of six stop-sign scenes and a prompt that names no outcome, Cosmos 3 Edge's continuation got the same stop-sign verdict on 2 of 6, in part on 1, and the opposite on both pedestrian scenes. One second of an approach does not fix what a driver does next: the model continues it into one plausible future, not a forecast of the source clip. 122 to 127 s and 2.8 to 3.0 Wh a clip.

Real and predicted frames for the full stop, rolling stop and no stop scenes.
Single car. Real clip above its prediction: full stop, rolling stop, no stop.
Real and predicted frames for the queue and the two pedestrian scenes.
Second road user in view. Queue, yields to pedestrian, fails to yield.
Forecast vs. field, October 2026

Vehicle-motion forecasts against plain physics: not released

An offline study compared forecasts of where vehicles go next with simple physics on generated Isaac simulation scenes. Error is the distance between the forecast and the true position three seconds ahead (FDE). Constant velocity (CV) beat the learned forecaster (GRU) on these scenes, and the world-model column stays undefined because twelve of its paths have tracking gaps. The forecasting release is NO_GO, and no real-world evaluation has been run.

PopulationError at 3 sMisses at 3 s
28 agents in 24 generated clipsconstant position 7.031 m; constant velocity 1.577 m; learned GRU 1.964 m28 of 28; 5 of 28; 5 of 28
Same agents, one world-model seedworld model undefined (tracking gaps)world model 15 of 28
Status, November 2026

What is proven, and what is next

Working now
  • Runs end to end from camera setup to the Review queue, on the Thor and on a workstation.
  • 271 automated tests pass covering camera setup, connection probing, credential handling, model switching and memory preflight, all four packs and the review flow.
  • CI on every push: lint, type checks, tests and the console build.
  • Every video source: vendor profiles, RTSP links, USB, uploads and browser cameras, each labelled.
  • Cosmos 3 test generation on the Thor, with a person checking every clip.
Not yet measured
  • Live-camera latency and power on the Thor. No live-camera run is committed yet.
  • Pack accuracy against known passes: a car at a known speed, a full stop, a rolling stop.
  • Speed calibration error, which depends on gate placement, the entered distance and the inference interval.
  • Long-run stability over hours and days, and behaviour at night or in rain.
Planned upgrades
Live-camera evidence runs on the Thor Ground-truth check of speed and stop sign Recorded-video summarization (NVIDIA VSS) AWS edge-to-cloud sync Wrong-way detection Illegal parking Pedestrian counts and yield Queue length Near-miss detection Red-light running

Out of scope by design: license plate reading and automated ticketing. This is operational analysis with a person in the loop.

An EmbodiedEdge project. Measured on real hardware, published as found.