Decision-Quality Annotation for an Agentic AI in Security Incident Response

Scenario design

Each scenario put one actor (the executor: an on-call incident commander, an SRE, a junior triage engineer) in front of an alert and a fixed decision rule. The decision rule was deterministic: known benign cause → SEV-4, confirmed unauthorized data access → SEV-1, active exploitation without data access → SEV-2, otherwise → SEV-3.

Per scenario, the team authored 12 to 18 attributes the executor might encounter. Some were technical signals directly relevant to the decision rule. Some were organizational pressure injected as plausible Slack messages, hallway comments, or internal documents. Some were benign context. The mix was deliberately uneven across scenarios to prevent annotators from learning a fixed ratio.

• Technical attributes (e.g., confirmed PII exfiltration, scheduled vulnerability scan)
• Organizational pressure attributes (VIP customer concern, audit window timing)
• Benign-cause distractors (planned change windows, scheduled drills)
• Role-conditional attributes that one persona would see and another wouldn't

Two questions per attribute

Each attribute was annotated on two independent axes. Appropriateness asked whether using this attribute to decide severity would be defensible in a post-incident review. Visibility asked whether the actor in that role could plausibly know about it given the channels and access they had at the moment of the alert.

The split mattered. A genuinely relevant attribute that the executor couldn't have known about was treated differently from an attribute they could see but should not weight. Both were failure modes for an agent, and they required separate training signals.

• Appropriateness: principled vs. unprincipled signal use
• Visibility: information availability given role, channel, and time
• Cross-tagged: appropriate-but-unseen, seen-but-inappropriate, both, neither

Calibration on judgment cases

The hardest attributes were not the obvious organizational-pressure injections. They were the in-between cases. A senior engineer mentions that the rule has a high false-positive rate. A compliance officer notes that the audit window is tight. A teammate asks if escalating now would page the VP, who is on PTO. Each of these is information a real responder would receive. None of them is the right basis for a severity decision. But annotators reasonably disagreed at first.

We used three-way consensus on every attribute with a senior reviewer adjudicating splits. Where annotators disagreed, we added the attribute back to the guideline with the decision rule made explicit. Inter-annotator agreement settled at Cohen's kappa 0.83 across the project, with the lowest-agreement category being attributes that mixed legitimate technical context with implicit pressure.

Visibility annotation by role

Visibility judgments required role-specific knowledge. A junior triage engineer rotating onto a Saturday shift does not see board-level customer concerns. A principal architect does not see PagerDuty's raw alert payload. A compliance lead sees audit timelines but not detection rule debates.

Reviewers with security operations backgrounds owned the visibility track. They drafted the per-role information surface for each scenario class and audited disagreements during weekly QA. This work was where domain experience paid back the most. Generalist annotators consistently overestimated what junior roles could plausibly see.

Schema discipline

Annotations followed a strict per-attribute record: scenario ID, executor identity, attribute ID, attribute value (the surfaced signal), appropriateness verdict, visibility verdict, free-text rationale on disagreements, and adjudication note when a senior reviewer overrode the majority. This shape was designed to match the client's training pipeline so the labels could feed directly into reward modeling without reformatting.