Structured Extraction From 50,000 Financial Documents for a Document AI Vendor

Stage 1: Layout segmentation

Annotators classified document regions using a schema adapted from LayoutLMv3 (Huang et al., 2022). Regions included header, body text, table, footer, signature block, and stamp.

Each region was outlined with a polygon, not a bounding box, because financial documents often have non-rectangular layouts (rotated stamps, overlapping signatures, marginal handwritten notes).

• Reading order tags for multi-column layouts
• Per-region rotation angle when content was off-axis
• Cross-page region linking for multi-page contracts

Stage 2: Entity extraction

Entity-level annotation followed the FUNSD form-understanding schema (Jaume et al., 2019), extended with a cross-page entity linking layer for contracts and multi-page statements.

• Entity types: counterparty name, amount, currency, date, account number, IBAN, tax ID
• Per-entity confidence flag for ambiguous cases (smudged numerals, partial occlusion)
• Semantic role labels distinguishing tax amount from subtotal from total

Stage 3: Table reconstruction

Table annotation referenced the schema published in PubTables-1M (Smock et al., 2022, Microsoft Research) and the cell structure approach from TableFormer (Nassar et al., 2022, IBM Research).

Borderless tables, nested tables, and tables rotated for fold-out printing were treated as separate edge cases with their own guideline pages.

• Row and column indices, including merged cells
• Header row vs. data row classification
• Continuation tables across page breaks linked by table ID

Quality assurance

Every document went through a two-pass review. Cohen's kappa stayed above 0.87 across the project. The domain lead audited a 10% random sample weekly and adjudicated all disagreements above an 80% IoU threshold for region polygons.