Scaling Multi-View Robotic Video Annotation From Manual Process to 1,000-Hour Ramp

Schema design for vision-language-action data

Label schema was designed against the client's downstream model architecture, drawing on the input format conventions in RT-2 (Brohan et al., 2023, Google DeepMind) and the Open X-Embodiment dataset standard (Padalkar et al., 2023). Action labels, object tracks, and spatial language were specified to feed the VLA training pipeline without reformatting.

• Action segments: start frame, end frame, action verb, target object, sub-action chain for compound motions
• Object tracks: per-instance track IDs, bounding boxes at keyframes every 10 frames with interpolation, attribute tags
• Spatial positioning language: relative-position phrases ("to the left of", "behind", "approaching", "in contact with") tied to the action grammar

Multi-view synchronized annotation

Multi-view robotic data introduces a coordination problem that single-camera annotation tooling cannot handle. The same world event has to be labeled consistently across overhead, wrist, and third-person views. Annotators reviewed all camera streams in a synchronized viewer with shared timeline scrubbing and cross-view event linking.

Spatial language labels were authored against the third-person canonical view but verified against wrist-view contact and overhead-view layout. Disagreements between views surfaced calibration issues and labeling errors that single-view annotation would have shipped.

Pilot week and calibration

The first week ran a representative pilot on 40 hours of video sampled across all task classes the November model needed to learn. The sample deliberately overweighted edge cases: occluded objects, low-light scenes, cluttered backgrounds, and contact-rich phases. Easy examples were not the bottleneck. Hard ones were.

Guidelines went through three revisions during week one. By Friday the schema was stable and inter-annotator agreement on action segments had settled at Cohen's kappa 0.86. Object tracking F1 against reviewer-authored ground truth came in at 0.91. Spatial language agreement, the most subjective track, was held to three-way consensus throughout the project.

Ramp to production throughput

Production batches scaled from 40 hours/week in week two to a steady 140 hours/week from week four onward. The team finished ramp two days inside the October readiness target with a 50-hour buffer for late client-driven schema additions.

• 30 video annotators on action and object tracks
• 8 reviewers with robotics or video understanding backgrounds owning QA and adjudication
• 2 project leads coordinating delivery, schema versioning, and client check-ins
• Daily QA sample of 5% of delivered video, full audit on any class showing IAA drift

Engineering time freed

The headline outcome the client cared about was not the annotation cost. It was the engineering hours that did not have to be spent labeling. Based on their estimated baseline pace before the engagement, this work would have absorbed roughly 3,000 engineer-hours had it stayed in-house. Those hours went back to model training and eval infrastructure.