Subjective Video Quality Scoring at 98% Agreement for a Generative Video Model Team

Five-metric rating schema

Every video was rated on the client's five quality dimensions, each on a 1 to 5 scale with anchored definitions written into the guideline. The overall score followed published anchor language so it stayed comparable across raters and across regions.

• Noise: low-level artifacts, grain, compression blocks, temporal flicker
• Sharpness: edge clarity, focus, detail preservation against denoising softness
• Exposure: brightness range, clipping, dark crush, scene-appropriate dynamic range
• Color: saturation, white balance, hue accuracy against scene context
• Overall: 5 excellent, 4 minor imperfections, 3 visible but usable, 2 multiple clear issues, 1 severely degraded

Pairwise A/B preference per metric

Beyond single-video ratings, raters compared A and B variants from the same generative prompt and chose a preferred variant per metric. This pairwise mode is what the client's reward model consumed. The methodology drew on RLHF-style preference annotation as used in InstructGPT (Ouyang et al., 2022) and recent generative-video preference work in VBench (Huang et al., 2024).

Reasoning capture was offered as an optional free-text field. About 40% of pairs came back with reasoning. The optional structure was intentional. Required reasoning produces shallow text that adds noise. Optional reasoning gets used by raters who genuinely have something to say, and those notes turned out to be the most valuable downstream signal.

Cross-continental rater pool

The 60-rater pool was deliberately distributed across the USA, UK, India, and Bangladesh. Color perception and viewing-condition norms vary by region, and a single-country rater pool would have produced a model that worked well for that region and drifted elsewhere. Distributing raters across four countries forced the consensus to cover the diversity the client's production traffic would see.

Each region had its own calibration sub-lead who knew the local conventions. Disagreements that traced to regional perception (saturated reds preferred in one market, more muted in another) were surfaced explicitly rather than averaged away, and the client used those signals to inform downstream model behavior.

Calibration anchors and pilot

The pilot week ran 400 anchor videos through every rater. Anchors had pre-determined target ratings authored by the client's image quality team. Raters whose scores drifted more than one step from anchors were retrained before joining production. Anchors were re-sampled every week during production to catch drift over time.

• Per-metric calibration thresholds with retraining trigger at >1 step drift
• Weekly anchor sampling on 5% of in-flight work
• Display calibration check at workstation setup
• Reference videos in the guideline for each 1 to 5 rating

How we reached 98% agreement

Subjective scoring at 98% inter-rater agreement does not come from any single fix. It came from stacking the right design choices.

• Anchored definitions per metric instead of free-text descriptions
• Pairwise preferences as the primary annotation mode, with single-video ratings as a secondary signal that pairwise verifies against
• Multi-region calibration to surface and handle regional bias instead of averaging it away
• Senior reviewer adjudication on any pair where regional sub-leads disagreed
• Iterative guideline revision driven by the optional reasoning corpus, which exposed where the schema was unclear
• Hard rejection of raters who failed weekly anchor re-checks