Measure an automation pilot by the exceptions it leaves behind
Count the items an automation completes correctly and the work required to repair the rest. A busy workflow dashboard can conceal a pilot that leaves the team with more checking, retries and unresolved cases.
Count the work left for people
Consider a fictional pilot that receives 100 requests. Its dashboard reports 100 completed workflow runs. In the work queue, however, 15 requests still need correction, five require someone to determine whether an action happened, and several have been sent to another inbox. The technical completion count does not describe completed business work.
Do not reuse those illustrative numbers as a benchmark. Their purpose is to show why a pilot needs a record at the level of the task, not just the automation run. One task can cause several runs, and one successful run can leave a person with substantial recovery work.
| Task record | What it reveals |
|---|---|
| Accepted result | Work completed to the agreed standard |
| Corrected result | Work completed with manual recovery |
| Unresolved result | Work still waiting for a decision or investigation |
| Repeated attempt | Additional execution, not necessarily additional value |
| Rejected input | A boundary of the process or its input quality |
Make the categories mutually understandable before collecting the numbers. For example, a corrected result should not also remain in the unresolved total unless the report explicitly tracks separate dimensions.
Give recurring exceptions a cause, not just a label
“Manual intervention” covers very different situations. A missing customer reference may require a lookup. An ambiguous instruction may require a decision. A duplicate action may require repair. Grouping them under one label makes it difficult to choose the next improvement.
During the pilot, record a concise reason for each exception and the work needed to resolve it. Avoid copying the full request when a category and a reference suffice. Where timing is relevant, define the start and end of the measured interval; otherwise two reviewers may count waiting time differently.
The n8n documentation on error handling can help organise technical failure paths. It does not decide whether an apparently successful output is acceptable to the business. Keep that acceptance decision visible in the task record.
Compare with the process the team would otherwise use
A pilot can look attractive when only its automated steps are measured. Include setup, review and recovery if the question is whether the team saves work. Keep those components separate so that a one-time setup cost is not confused with recurring effort.
Use comparable tasks when examining the old and new processes. If the pilot receives only easy cases while people retain the difficult ones, report that selection. It may be a sensible first deployment, but it limits conclusions about the entire workload.
At the review meeting, bring examples from the recurring exception categories. A count tells the team where time may be going; a representative record explains what has to change. The next iteration can then target a missing input, a weak approval step or an uncertain recovery path instead of increasing volume in the hope that the process improves by itself.
Keep a small outcome ledger
The categories should be mutually understandable to the people doing the work. Adapt their names to the process, but avoid merging unresolved cases into completed work.
Use the same outcome definition when comparing automation tools, as discussed in our French guide. Otherwise, each tool's dashboard may count a different kind of success.
Measure recovery where it happens
If you want a time estimate, record actual review and repair time during the pilot. Do not subtract an assumed manual duration from every run.
Capture the reason for intervention: missing data, misunderstood rule, unavailable service or an action requiring a person. These reasons reveal whether the solution needs a better input, a corrected rule or a deliberately retained human step.
An error workflow can notify the right person. The n8n error-handling documentation describes available mechanisms, but your team must decide what recovery means for each failure.
Decide what the team can operate next
Compare the recovery work with the team's available capacity. An automation that clears its queue by moving unresolved cases into another inbox has not removed that work.
State what evidence would change the decision. A corrected input source or a tested recovery path is a more useful condition than “try again when the tool improves”.
End the pilot review with a decision tied to the remaining work: expand within the tested scope, repair a recurring exception, or narrow the process. Assign the open cases before increasing the volume.