Video QC Bench

Can video language models find continuity and physics defects in AI-generated video? Five systems, each run as five specialist forensic observers plus a generalist baseline over 14 test clips, scored against a human reviewer's defect baseline. Specialists track narrow invariants (object persistence, character continuity, contact mechanics, spatial topology, causal order) instead of answering a broad “find any issues” prompt.

Leaderboard: detection credit vs false-positive penalty

Detection credit sums priority weights (P0=8 … P4=1) over unique human-baseline defects a model found; each false-positive finding costs 1 point. Findings that are plausible but absent from the human baseline (“extras”) are neutral.

Cost of the full run vs detection

84 calls per system (14 videos × 6 roles). Cost is the metered API spend for the whole run; Pegasus 1.5 is estimated from TwelveLabs' $0.021/analyzed-minute pricing. The efficient frontier is the upper-left.

What each system's findings were made of

Every reported finding was adjudicated: a true positive matching the human baseline, an “extra” (plausible defect the human didn't list — several were verified real on frame review), or a false positive.

Defect coverage matrix

All 36 human-baseline defects. Filled cells are detections; darker means higher priority. Hover any cell for the defect description.

Where the specialist roles earn their keep

True-positive findings per role. The generalist baseline is the control: for Gemini Pro Latest, 11 of its 20 detected defects came only from specialist passes.

Missed by every model

    Model-found extras

    Plausible defects reported by the models but absent from the original human baseline.

      Method & caveats

      Generated · scores & raw model output in the repo · weights P0=8, P1=5, P2=3, P3=2, P4=1, FP penalty −1 · built with hand-rolled SVG, no dependencies.