# Build an Agent Regression Set Before Adoption

> Source: The Agents Index — https://theagentsindex.com/guides/agent-regression-set-before-adoption/
> By: The Agents Index Editorial, Research desk
> Claims last verified: 2026-09-12
> Last updated: 2026-09-12
> Covers: Building a reader-owned evaluation set before agent adoption, Recording known-good outcomes, judgment criteria and reviewers, Comparing resolution, escalation, human effort, time and cost, Retaining task-level failures and run evidence, Separating capability failures, permission failures and ambiguous tasks, Maintaining frozen baselines alongside evolving task collections, Interpreting hypothetical support results without claiming productivity gains, Reviewing framework references monthly

Freeze real tasks, define who judges success, and compare outcomes, escalations and failures before and after an agent enters your workflow.

## Measure the same work, with the same rules

Before adopting an agent, save a small, fixed set of real tasks, their starting conditions, and the criteria for an acceptable outcome. Run your current workflow against that set and retain its outputs. After adoption, run the unchanged set through the agent-assisted workflow. Compare task-level outcomes, human involvement, time, cost and failures, not just one success percentage.

This is a documentation-based walkthrough. We did not exercise account-level steps, and no pilot was run for this guide. The evaluation procedure below is our editorial recommendation; the numerical example is explicitly hypothetical.

Source-check date: September 12, 2026. Review this guide monthly, with the next review due October 12, 2026, and update its framework references and affected recommendations when the cited frameworks change.

## What the frameworks contribute

[NIST’s Playbook](https://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook) aligns its suggestions with four AI RMF functions: Govern, Map, Measure and Manage. NIST says the Playbook is neither a checklist nor a set of steps to follow in its entirety. Use that structure to organize ownership, context, measurement and decisions, rather than treating this tutorial as a compliance checklist.

[OWASP describes its agentic AI guide](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/) as a threat-model-based reference of emerging agentic threats that discusses mitigations. That threat-and-mitigation framing belongs to OWASP; the practical failure labels below are our triage convention, not an OWASP taxonomy. This walkthrough uses source-page summaries and does not reproduce or claim to implement a detailed threat taxonomy. Use frameworks as a structure for reasoning, not as a certification of your agent or pilot.

## 1. Collect tasks before seeing agent results

Choose a manageable batch from your own work using a selection rule written in advance. For example, take consecutive eligible tasks from a defined period, then add separately labelled cases for costly mistakes, required handoffs and unclear requests. Record the population and exclusions. Do not quietly select only the tasks that look easy to automate.

For each task, preserve the original request and the relevant starting state: documents, policy version, repository revision or a redacted ticket snapshot. Define what tools and data the workflow may use. Use an isolated copy or replay environment for tasks that would otherwise change customer or production state.

Write a known-good outcome before the agent attempts the task. This need not be one exact answer. It can be a set of acceptable answers, a required state change, a correct refusal or a justified escalation. If an old outcome is merely what happened, have a reviewer establish whether it was actually acceptable.

Keep genuinely ambiguous cases visible. Give them a criterion such as “asks for the missing account identifier before acting,” rather than inventing a definite answer after the run.

## 2. Make the task record judgeable

Use a spreadsheet plus an evidence folder, or an equivalent versioned record. Our recommended minimum fields are:

| Field | What to preserve |
| --- | --- |
| Identity and scope | Task ID, set version, selection rule, task type and any exclusions. |
| Frozen inputs | Request, starting-state snapshot, supporting material and policy versions. |
| Known-good outcome | Acceptable result, required evidence and prohibited actions. |
| Judgment rule | Pass/fail criterion, rubric author, outcome judge and how disagreements will be resolved. |
| Allowed execution | Tools, permissions, human assistance, time limit and retry budget. |
| Run identity | Date, workflow, agent/model version where available, instructions and configuration. |
| Run evidence | Output, relevant action log, handoffs, corrections, time and recorded charges. |
| Decision | Pass/fail, judge, rationale, failure label and any uncertainty. |

Define the retry rule before the baseline. If one workflow gets three attempts, do not compare its best attempt with the other workflow’s single attempt without labelling that difference. Record all attempts and the rule used to choose or combine their results.

## 3. Record a baseline with paired measures

Run the existing workflow on the frozen tasks. If a historical result lacks its inputs, timing or judgment evidence, mark those fields unknown rather than reconstructing convenient numbers.

Predefine each measure and its denominator. For a support team, our recommended scorecard is:

- Accepted outcome: tasks meeting the written criterion divided by all tasks in the set. A correct escalation may pass even though it is not a resolution.
- Resolution: tickets meeting a specified resolution criterion within a fixed observation window divided by all tickets in the set.
- Escalation: tickets handed to another person or team at least once divided by all tickets in the set. Record why each handoff occurred.
- Human involvement: review, correction and handoff work, measured separately from the agent’s activity. Define whether the initial operator counts.
- Time: elapsed time to an accepted outcome and active human minutes, recorded separately. Keep unfinished tasks visible rather than reporting only completed ones.
- Cost: recorded service charges and human time shown separately, or combined only with an explicitly stated labor-cost assumption.
- Safety and rework: prohibited actions, incorrect changes, repeated attempts and reopened work within the chosen window.

Also write an adoption rule before seeing results. For example, require improvement on your chosen outcome measure without exceeding your own limit for human effort or unacceptable actions. Set those limits from your operational requirements, not from a universal productivity target.

## 4. Run the unchanged set after adoption

1. Restore the same task inputs and starting conditions. Apply the same rubric, observation window and retry rule.
2. Record the agent-assisted workflow’s configuration and actual permissions. Document unavoidable differences from the baseline.
3. Have the designated reviewer judge each outcome. Where practical, hide which workflow produced it; retain the reviewer’s rationale and any adjudication.
4. Keep outputs and logs for failures, timeouts and abandoned tasks as well as successes. Do not rerun away an inconvenient result.
5. Compare each task with its baseline counterpart, then summarize counts, rates, human effort and safety outcomes.

Attribute the comparison to the workflow you actually evaluated. If you changed the knowledge base, permissions and staffing alongside the agent, describe a workflow change rather than claiming the agent alone caused the difference. For a focused follow-up, change one recorded condition at a time.

## Worked example: more resolutions, more escalations

Hypothetical arithmetic only: suppose a support team replays the same 20 tickets before and after adoption. It defines resolution as meeting its ticket-specific criterion by the end of the observation window, including resolutions completed after a handoff. Escalation means any handoff to another person or team. The counts below are invented to explain the comparison, not findings from a pilot.

| Ticket outcome | Baseline | Agent-assisted workflow |
| --- | --- | --- |
| Resolved without escalation | 10 | 10 |
| Resolved after escalation | 2 | 6 |
| Unresolved after escalation | 2 | 2 |
| Unresolved, no escalation | 6 | 2 |
| Total tickets | 20 | 20 |
| Resolution rate | 12/20 = 60% | 16/20 = 80% |
| Escalation rate | 4/20 = 20% | 8/20 = 40% |

In this hypothetical example, resolution rises by 20 percentage points while escalation also rises by 20 percentage points. Resolutions without escalation remain at 10. Reporting only “resolution reached 80%” would hide the additional handoffs and would not establish reduced human work.

Our interpretation: this result would justify examining the handoffs, not automatically accepting or rejecting the agent. The extra escalations might be appropriate routing or avoidable work for specialists. Review their reasons and human minutes before deciding. The table alone establishes neither a productivity gain nor a cause for the change.

## 5. Keep three failure diagnoses separate

Use these local triage labels to decide what to investigate. They are not framework-defined risk categories.

| Label | Hypothetical case | Recommended response |
| --- | --- | --- |
| Capability failure | The request and policy are clear, the required access works, but the agent calculates the permitted refund incorrectly. | Keep the failed output and rule it violated. Investigate the agent’s reasoning or execution under the recorded conditions. |
| Permission failure | The required refund action is denied because the workflow lacks the necessary permission. | Record the denial and intended access policy. Assess deployment readiness separately; do not infer reasoning ability from an action the system could not perform. |
| Ambiguous task | The request omits which of two orders should be refunded, and the original rubric never defined clarification behavior. | Mark the task or rubric ambiguity. Preserve the original result and repair the case in a new set version, not in place. |

Only the first diagnosis directly concerns the agent’s ability to complete a clear, permitted task. The other two identify access or task-definition problems first. Still judge behavior at the boundary: if the rubric required clarification and the agent guessed, that is an agent failure; if it attempted a prohibited workaround after a denial, record that action separately.

Do not erase permission-blocked tasks from the operational score. Keep the overall completion count and show the blocked subset alongside it. If you also report a capability-only score, state its narrower denominator. Leave uncertain cases unresolved until the evidence supports a diagnosis.

## 6. Freeze the comparison; version the improvements

Keep the original task set, rubric and selection rule immutable. Editing tasks after seeing results destroys the comparison: the revised result no longer answers how the two workflows performed on the same work.

When you discover a bad reference answer, preserve it with an erratum. Create a corrected set version and run both workflows against that version before making a new comparison. Do not compare the old baseline with the corrected agent score.

Maintain a second, explicitly changing collection for new production failures and emerging tasks. Use it to develop future evaluation versions, but do not merge its scores into the frozen baseline. Record which cases were used for tuning; reserve separate unseen cases when you need evidence beyond the familiar regression tasks.

For every adoption or later release decision, retain the set version, run configuration, paired results, failure records, judges and decision rationale. That package makes the observed difference inspectable. A small selected set does not, by itself, establish performance across all future work or isolate a causal productivity effect.

## Maintenance and evidence limits

[NIST’s Playbook page](https://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook) says AI RMF 1.0 is being updated and that the Playbook will be updated after the framework is revised. Treat this retrieval as the guide’s baseline observation, not evidence of a newly announced change. At each monthly review, check for framework revisions, reassess the attributed wording and record any substantive correction.

The cited source pages establish high-level risk-management framing, not this exact evaluation protocol, a sample-size guarantee or a productivity expectation. The task schema, metrics and worked example here are editorial recommendations for making your own evidence reviewable.

## Sources

- [OWASP: Agentic AI – Threats and Mitigations](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/)
- [NIST AI RMF Playbook](https://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook)
- [NIST: AI Risk Management Framework overview](https://www.nist.gov/itl/ai-risk-management-framework)

## FAQ

### What if we already adopted the agent without collecting a baseline?

Freeze a task set now and, where feasible, replay the previous workflow against the same starting conditions. Label that as a reconstructed comparison, not a contemporaneous pre-adoption measurement. If the old workflow cannot be reproduced, use the current run as the baseline for future changes and leave the earlier improvement claim unproven. Mark missing historical timing or judgment evidence as unknown.

### Can we tune the agent using the frozen regression tasks?

Yes, as a development choice, but record that those cases were used for tuning. Keep their inputs and scoring rules unchanged so they remain useful for regression comparisons. For a broader adoption decision, our recommendation is to add a separately versioned, unseen set rather than treating performance on familiar tasks as evidence for all future work.