Ember Compute / Projects / Janus
Project · Research runtime
Janus
A video world model that learns and infers at the same time, on the same machine, against the same live stream.
Eighty-five seconds, then a real run at 8× speed
The first half is the architecture. The second half is a real 300-second run on the synthetic world: the regime changes at 45 seconds, surprise spikes, and falls again as the learner catches up while the inferencer keeps serving.
Every number in the animation is illustrative, shaped like the measured v0 runs. The video's second half is a recorded run, not an illustration.
Two faces, one set of weights
Janus is named for the two-faced god, because that is the architecture. One face looks forward, predicting what the next half second holds. One looks back, replaying what already happened. They share a head: the same weights, handed from the learning face to the serving face while the stream keeps running.
One process serves predictions at frame rate. A second process trains continuously on the same frames and publishes fresh weights onto a shared-memory bus. The server swaps them in mid-stream without dropping a frame. Nothing is labelled by anyone. What the model serves at time t becomes its own training label at t + 0.5 s, because the future frame arrives and settles the bet.
It predicts embeddings, not pixels. Pixel prediction spends all of a model's capacity on detail that is inherently unpredictable. Latent prediction spends it on structure that is not. The target is a slow-moving copy of the model's own encoder, so the whole loop is self-supervised.
The served output is surprise: how far the frame that actually showed up sits from the prediction made half a second earlier, scored against a running baseline. Surprise falls as the model learns a scene and spikes when the scene does something new. That makes it an unlabeled, on-device novelty signal, and the reason the project exists.
Adaptation as a compute-efficiency problem
Every robot and camera that ships today is frozen at the factory. Re-training a fleet is a datacenter job. Janus is the other path: keep adapting on the device, with a model small enough that the cost of staying adaptive is paid in parameters, not in racks.
Small on purpose
The v0 world model is 2.7 million parameters on 96 × 96 frames. Learning adds roughly three times the weight traffic of serving on the trained subset, so smallness is what makes simultaneous learn-and-serve fit on one consumer GPU, or one edge module.
The runtime is the IP
The two-process loop, the weight bus, the replay reservoir, scoring each prediction in the weight version that made it, and the replay-ratio throttle. The world model inside is replaceable. The loop is not.
Measured on the frame thread
Every latency number is p99 on the serving thread with the learner running beside it, and every quality number is reported against a frozen control and a no-replay ablation on the identical stream.
Four things the literature did not hand us
Each of these was found by instrumenting the system, not by reading about it. They only exist because learning and inference are simultaneous.
The measurement itself is non-stationary
A prediction is made in the embedding space of weight version V and scored half a second later, by which time the learner has usually published V+1. Score across that swap and much of your "surprise" is the model's own weight drift, not novelty. In a default run about 83% of frames are scored across a swap.
Fix: keep the last few target encoders and always score a prediction in the coordinate frame that produced it. It never crashes. It just quietly corrupts the signal.
A fast learner memorizes the buffer
Left uncapped, the learner takes about 120 steps per second against a 30 fps stream, so every frame is replayed about 100 times before it ages out. Training error collapses to 0.07 while served error climbs to 0.64. A triumph in the training log, a failure on the wire.
Fix: clip-consistent augmentation, a slower target, and a replay-ratio throttle. Before the fix, injected anomalies were indistinguishable from normal frames. After it, they separate cleanly.
Raw prediction error is not a result
An untrained encoder is very smooth, so consecutive frames embed almost identically and its raw error looks excellent for a degenerate reason. Every number is therefore reported as skill against persistence, the "future looks like now" baseline that operational weather forecasting has used for decades.
Also: the cheapest way to zero prediction error is to emit a constant. Embedding effective rank is logged every second as the collapse tripwire.
The weight copy was never the bottleneck
The first version of the loop showed a 15 ms p99 spike on every swap. The 11 MB weight copy was blamed. Isolated, it cost 0.5 ms. In situ, both processes ran the framework's default 16 CPU threads on a 16-core box and the frame thread kept getting descheduled.
Fix: cap threads per process, stage new weights off the frame thread, and make the model's parameters views into a device double-buffer so the swap is about fifty pointer rebinds. Swap p99 went from 15 ms to under 3 ms. On a six-core edge module the lesson gets sharper, not softer.
Live, versus a frozen control, versus no replay
Three configurations run against the identical deterministic synthetic stream. The world rotates its generative regime every 45 seconds and injects brief out-of-regime events, which is what makes novelty measurable rather than anecdotal.
| Metric | Live | Frozen control | No replay |
|---|---|---|---|
| Served frame rate | 29.95 fps | 29.95 fps | 29.95 fps |
| Inference, p50 / p99 | 1.67 / 2.67 ms | 1.75 / 1.96 ms | 1.60 / 2.38 ms |
| Gradient steps while serving | 26,941 | 0 | 26,946 |
| Weight versions served | 539 | 1 | 539 |
| Skill vs persistence | +0.246 | −15.60 | +0.194 |
| Novelty AUC | 0.872 | 0.451 | 0.811 |
| Probe error on early clips (forgetting) | 0.531 | — | 0.802 |
300 seconds and 9,000 frames per configuration, two consumer GPUs. Skill is 1 minus the ratio of prediction error to persistence error; positive means the model beat "the future looks like now". AUC is area under the ROC curve against ground-truth anomaly labels.
Real video
One fixed camera, 20 minutes at 25 fps, one GPU time-sliced between both faces, 1,839 weight versions served. Novelty AUC 0.70 live against 0.60 untrained, averaged over 19 labelled clips. Prediction skill only +0.05: on a near-static scene, persistence is a strong baseline and a 2.7M model barely beats it. Learning helps detection more than prediction.
Retention
After nine minutes alone with a sixth regime, the model with replay returned to the five old regimes at +0.06 above where it left them. Without replay it lost three of the five, at −0.37 on average, and then lost the sixth again when the old ones came back.
Weight hand-off
A device-side weight bus against the host bus was a latency null: 0.4 ms swap cost either way. What jitter remains is GPU time-slicing between the two faces, which is the next lever and the one that matters on a single edge GPU.
Where it is going
An on-device continual-learning sidecar. The large labs are solving "learn to do" for robots. Nobody is solving "keep adapting in one building for three years without a fleet retrain." Janus sits under whatever policy a robot ships with, watches the same stream, and keeps a current model of what normal looks like here.
Surprise as the safety primitive. An unlabeled, on-device "I don't know" is the signal a deployed system needs before it acts, and the one that is hardest to fake.
The next plot. Served frames per second against gradient steps per second against watts, on a single time-sliced edge GPU, to find where the serving budget breaks. That curve decides the product.
Edge robotics, on-device adaptation, or a stream that never stops.
The other half of the thesis: serving a mixture-of-experts model bigger than any one device, across the devices on a wireless network.