Automatic XFAIL management: from a failing test to its root cause in a minute

The challenge

At the client, test results drive release decisions, and the test suite grew in size and complexity along with the product. New versions could fail regression tests for reasons that were already known, and those failures blocked ongoing development. Marking such tests as XFAIL (expected to fail) kept work moving. But every XFAIL needed deep analysis by the test owner, slow back-and-forth between teams, and reruns of resource-heavy tests.

What was at risk:

• Blocked development efforts and growing costs.

What we did

• Make XFAIL marking automatic and deterministic, driven by root-cause analysis and test results, so the same failure always gets the same decision.
• Fingerprint root causes and test cases, so recurring failures are recognized and marked without fresh analysis.
• Use a Python scanner with AI-assisted root-cause analysis to route each failure to the right test owner.
• Build a web application where XFAIL candidates are collected, categorized and confirmed by test owners, so a person still signs off.
• Add an observability service across the data and communication layers, so the process can be tuned to what DEV and QA teams need.
• Track statistics on failures and XFAILs and feed them back into the client’s development process.

How we built it

A known test failure can move from a raw run result to a tracked expected failure through three mechanisms: a tag in the test definition, a manual declaration in CAT (commit approval tool), or a candidate surfaced by DB analysis. They share history and reporting, and a scan candidate is evidence for a person to review.

1. Collect Jenkins schedules pyATS regression runs. At completion, the reporter sends run status, console information, and summary XML to CAT. The XML records test case outcomes and metadata; a test framework tag can mark an expected failure. CAT business logic stores run and testcase history, while CAT testcase results are also available in DB.
2. Analyze Classification is deterministic: the system maps outcomes, applies active XFAIL declarations, and reconciles testcase results with suite counters. Only actual failures covered by an approved expectation can be exempted; errors, aborted, and blocked outcomes stay as they are. Unexpected passes remain visible for review. Separately, CAT queues a best-effort AI root-cause analysis for failed or crashed runs; XFAIL eligibility is decided by the rules above.
3. Detect and group The DB scanner groups recent CAT results by suite, testcase and normalized version. Candidates need at least five decided outcomes, with at least 70% of failures occurring in otherwise healthy suite runs. They qualify through either an 80% failure rate or at least one pass/fail transition. These thresholds are heuristics: history helps reviewers assess a candidate.
4. Review The dashboard shows candidates with failure rate, isolation, run history, version, testbed, and linked bug information when available. A maintainer reviews the evidence and creates a declaration with suite, testcase, optional version, bug ID, reason, and expiry. Review is manual: a candidate changes test status only once a declaration exists.
5. Declare and apply CAT business logic stores manual declarations and applies an active declaration to later CAT results when suite, testcase and version match, leaving test sources untouched. Teams can edit, delete, or expire a declaration; they can also maintain XFAIL tags directly in pyATS test plan files. Completed run judgments remain historical when declarations change.
6. Observe and act The dashboard combines provider, bug, and run-history details, including failure rates, transitions, last pass, and last failure. CI and reports use the reconciled suite verdict. Test maintainers, developers, and release teams use the evidence to prioritize fixes, review expectations, and check behavior after a fix.