The full build
Street cameras are set up in a web console. A vision-language model on the device looks at each frame, use-case packs turn what it sees into traffic events, and an operator reviews them. The full build adds the ways video gets in, ground-plane tracking, cloud delivery that survives outages, and Cosmos 3 runs that generate test events. Processing stays on the device, and nothing is enforced automatically. The source of the full build is not public; the open core is.
The video walks through the system on the Thor. The clips below are cut from screen recordings of the console on 2026-11-04. Each one plays an uploaded generated clip through a local CPU demo detector, and the console labels every frame "Uploaded · Generated, not a live camera".
Paste an RTSP link with its credentials taken out and stored encrypted, pick a USB camera with its sizes and frame rates listed, upload an MP4, MOV or MKV, or share a phone or laptop camera from the browser. All of them run through the same packs and review.
Each camera, event and run records where its video came from: live RTSP, USB, browser, uploaded recording, uploaded generated footage or synthetic. Anything that is not a live camera carries a visible label on Live, Events, Review and in the run record.
An optional per-camera add-on projects vehicles onto a calibrated flat road and estimates position, trajectory, speed in m/s and mph, heading, and a one to two second motion prediction. The packs and their events keep their own behaviour.
Optional optical flow measures how image features move inside each vehicle box and checks it forward and backward. Weak texture or a gap gives "unknown", never a zero speed.
Messages wait in a bounded outbox on disk until the broker acknowledges them, with retries across restarts. When the quota runs out the newest messages are rejected visibly, not silently.
Old or unprojectable measurements are withheld, speed warm-up restarts after a gap, and Live shows motion freshness, usable speed coverage, cloud backlog and delivered messages.
Each pack works on geometry the operator draws in the Studio page over a live snapshot from the camera. Coordinates are stored normalized to the frame, so they survive a change of stream resolution.
Watches each vehicle track through the stop zone and calls it a full stop, a rolling stop or no stop from its dwell time and slowest speed in the zone.
Times a track from gate A to gate B and converts the distance the operator entered into km/h or mph. Only measured crossings produce a reading.
Counts each tracked vehicle once as it crosses the A to B line, by direction and by class: car, truck, bus, motorcycle. Hourly totals per camera.
Tracks people or other chosen classes with direction and a short descriptor, across the whole frame or inside a drawn zone.
Speed and stop sign are refused on the same camera. They need different sight lines, so the console blocks the combination instead of producing weak results for both.
One capture thread per camera writes the newest frame to a single slot.
The browser reads that slot as MJPEG at the camera's own frame rate.
Inference samples the slot every interval. vLLM, Ollama or NVIDIA NIM, switchable live.
Detections get stable track ids, then each pack bound to the camera applies its rule.
Events persist in SQLite. Flagged ones wait in Review with the frame the model saw.
The model server is not in the video path. If it stops or falls behind, the Live page keeps playing and shows "inference unavailable" instead of inventing events. An end-to-end runtime test checks exactly that.
The defaults below are read from the pack code. Every one can be changed per camera in Studio.
Screens from the running app. These captures use the built-in synthetic test feed and the mock detector, and the console labels both on screen so they are never mistaken for a real camera.
Camera passwords are encrypted at rest with a key file outside the repository. The API never returns one, and a saved password is only reused for the host and port it was saved for.
Stream URLs are masked in logs, errors, status payloads and run artifacts.
On first start the console picks a default for the hardware: Cosmos-Reason2-2B on Thor, Gemma 4 on a workstation. The operator can switch at any time.
A model container another app depends on cannot be stopped by accident. Stopping it needs an explicit, logged confirmation.
A systemd unit starts the API and console at boot and keeps a persistent log.
Packs flag, operators confirm or dismiss. Each item can carry a ground-truth note, such as "my car, full stop", so pack output can be checked against known passes.
Rare, dangerous events cannot be filmed on purpose. Cosmos 3 Edge (4B) draws them from a still on the Thor, and the production packs score the result. Generated footage throughout: offline test data, not live prediction, and no real-world accuracy.
| Event | Prompt as written, then upsampled | Hand-written prompt | Hand-written prompt and a reference frame |
|---|---|---|---|
| Pedestrian steps out from behind a parked pickup | Partly | Yes | |
| Car rolls through the stop | Partly | Yes | |
| Car runs the stop with cross traffic | No | No | Yes |
Checked by a person watching each clip. The framework's prompt upsampler had turned "never stops" into "stops precisely at the STOP line". Hand-written prompts with compass directions, exclusions and timing fixed the single-actor events. The cross-traffic event also needed the second vehicle placed in the starting frame, because an actor that exists only in words is placed wherever the model chooses. Each clip took 114 to 125 s on the board at 2.6 to 2.8 Wh.



Given only the first second of six stop-sign scenes and a prompt that names no outcome, Cosmos 3 Edge's continuation got the same stop-sign verdict on 2 of 6, in part on 1, and the opposite on both pedestrian scenes. One second of an approach does not fix what a driver does next: the model continues it into one plausible future, not a forecast of the source clip. 122 to 127 s and 2.8 to 3.0 Wh a clip.


An offline study compared forecasts of where vehicles go next with simple physics on generated Isaac simulation scenes. Error is the distance between the forecast and the true position three seconds ahead (FDE). Constant velocity (CV) beat the learned forecaster (GRU) on these scenes, and the world-model column stays undefined because twelve of its paths have tracking gaps. The forecasting release is NO_GO, and no real-world evaluation has been run.
| Population | Error at 3 s | Misses at 3 s |
|---|---|---|
| 28 agents in 24 generated clips | constant position 7.031 m; constant velocity 1.577 m; learned GRU 1.964 m | 28 of 28; 5 of 28; 5 of 28 |
| Same agents, one world-model seed | world model undefined (tracking gaps) | world model 15 of 28 |
Out of scope by design: license plate reading and automated ticketing. This is operational analysis with a person in the loop.