Book a call

Test and evaluation infrastructure for physical AI.

Robots should fail in sim, not in the field. Made for teams building robots faster than they can test them.

Book a call
autonomy scenariosperception scenariosmanipulation scenarios

Each row is a commit. Each column is a scenario. Bars are scores; red is a regression.

AUTONOMY · LOCALIZATION

Every autonomy build,
tested before it ships.

Drive the same routes in sim or replay your logs, and compare pose error, clearance and goals reached with your last good build.

$ invariant eval --stack nav_1.4.3 --compare 1.4.2
planned routetrue pathlocalization error
5 m
SIM · warehouse_nav_0400:00 / 01:47
metric1.4.31.4.2
localization RMSE–0.04 m
min clearance–0.43 m
goals reached0 of 99 of 9
thresholdRMSE ≤ 0.10 m
running
Illustrative example
PERCEPTION · DETECTION

Catch regressions
before the field does.

Score every detection against ground truth on the same labeled scenes, and block the merge when one regresses.

$ invariant eval --model detector_v7 --compare main
front_camera · aisle_day_01detector_v7
person 0.94 · IoU 0.87 pallet 0.89 · IoU 0.82 crate 0.91 · IoU 0.78
┄ ground truth  ━ detection  ┄ missedIllustrative example
scenes0/50
modelv7 vs main (v6)
metric · 3 objects/scenev7main
recall–1.00
low-light recall–1.00
mean IoU–0.80
false positives–0
passing
scoring against ground truth…
MANIPULATION · POLICY EVAL

Grade every policy
before it grabs anything.

Run hundreds of trials per checkpoint, scored per object, so a better average can't hide a new failure.

$ invariant eval --policy pick_v12 --compare main --trials 500
pick_and_place · tabletop_simlast 25 trials per object
objectv12vs main
grasp + place  dropped / missedIllustrative example
trials · 100 per object0/500
policyv12 vs main (v11)
metricv12main
overall success–87.4%
mug success–88%
collisions–0
running
running trials…
HOW IT WORKS

From your laptop to every release.

01

Connect

Your autonomy stack, model or policy, with ROS 2 and MCAP logs.

02

Run

The same scenarios every time, simulated or replayed, with ground truth.

03

Compare

What got better or worse, on every commit and before release.

Looking for a few teams to pilot with.

We're building Invariant with a small group of design partners across autonomy, perception and manipulation. If testing can't keep up with how fast you ship, let's talk.

Book a call

or email founders@invariant.run