About Agentic QA
Agentic QA runs your regression test cases the way a manual tester would: it opens your site in a real browser, does what each step says as an ordinary user with a mouse and keyboard, and checks whether the page shows what the step expects. This page covers how a step is performed and judged, what each outcome means, and where the edges are.
Not an accessibility audit
Section titled “Not an accessibility audit”b8e audit and b8e qa audit both walk journeys in a browser, but
they answer different questions. An accessibility audit drives a screen reader and asks whether a
user of assistive technology can get through. Agentic QA asks whether the product does what the
test case says it does: the right page opens, the price is right, the button is disabled when it
should be.
The two keep separate journeys. A QA journey is a list of exact steps with expected results, usually brought in from an existing manual test suite. Its Findings are product defects, kept apart from accessibility violations in the dashboard’s Agentic QA section.
A tester and a judge
Section titled “A tester and a judge”Each step goes through two parties that never share a role.
- The tester performs the step. It sees the page as an accessibility view, with a screenshot when it needs one, and acts on it one action at a time, the way a person with a mouse and keyboard would. It does only what the step says. When a step names the way to get somewhere, such as a footer link, it takes exactly that way, because the way is what is under test. It never decides whether the product is right.
- The judge checks the expected result against the page the tester left: a screenshot, the accessibility view, and the actions the tester took. It reads values literally. A listed set of items holds when every item is there, unless the expectation says “only” or “exactly”.
Keeping them apart stops the party that did the work from grading it. A tester that also judged would be tempted to work around a broken control until the expectation held.
What each step outcome means
Section titled “What each step outcome means”| Label | What happened | A Finding? |
|---|---|---|
PASSED | The judge found the expected result holds. | no |
FAILED | The judge found it clearly does not hold, and says what the page shows instead. | yes |
INCONCLUSIVE | The judge could not decide, or a failure was held back because something else may explain it (see below). | no |
COMPLETED | The step had no expected result, and the tester did it. | no |
BLOCKED | The tester could not do the step: what it names is missing, the page errored, an action had no effect twice, or the step led off the site. | no |
AGENT ERROR | The tester or the judge failed to produce an answer. | no |
INFRA ERROR | The browser could not be observed. The journey stops here. | no |
SKIPPED | The step never ran, because the journey stopped before it. | no |
Only FAILED is a Finding. PASSED and COMPLETED are the clean ends of a step. The rest say
the step could not be checked, and that is a different problem: it needs a better step, a reachable
page or a retry, not a bug report. This is why b8e qa audit exits 1 only on a Finding and 8
when steps could not be checked.
Why a failure can come back inconclusive
Section titled “Why a failure can come back inconclusive”A Finding claims the product is wrong, so the judge only makes that claim when the page is the
likely cause. A step is INCONCLUSIVE rather than FAILED when:
- the judge’s evidence was thin or its confidence low;
- the expected result is a matter of taste or cannot be seen on a page;
- the step landed on a sign-in page it did not ask for;
- an action in the step went wrong and was not recovered;
- the step opened a new tab, which Agentic QA closes because it works in one tab;
- an earlier step was blocked or hit an agent error, and may have left the page in the wrong state.
In each case the reasoning names the cause, so a reader can tell a weak step from a real defect.
A blocked step does not stop the journey. The next step starts from wherever the page is, and is blocked only if it cannot be done from there. Only an infrastructure error ends a journey early.
A disabled or read-only control is the page’s answer, not an obstacle. The tester records that the control refused and moves on, and the judge weighs it: it fails an expectation that needed the action and supports one that says the control is disabled.
Site errors are not Findings
Section titled “Site errors are not Findings”While a journey runs, the browser also records what failed on the site underneath: server errors, failed requests and console errors. The report lists them and they are stored on the run, but they never become Findings. A page can log an error and still do what the test case expects, and a Finding is reserved for the expectation.
Why wording decides what can be checked
Section titled “Why wording decides what can be checked”A judge that grades from a screenshot and the page’s structure can only confirm what is on the page.
An expectation of taste, such as “looks good”, has no such evidence, so it is never PASSED
however good the page looks.
The tester is literal for the same reason the judge is: a test case is a claim about one control and one route. If the tester swapped in a similar-looking element, or reached a page by search instead of the footer link the step names, a pass would no longer say anything about what the step describes. It blocks the step instead.
What the tester will not do
Section titled “What the tester will not do”It never enters real credentials, payment details or personal data, and never submits a form unless the step says to. Navigation that leaves the site under test is refused and the step is blocked.
A journey runs signed out unless it runs as a persona, a test account whose login was captured with
b8e profile create. A step that lands on a sign-in page
it did not ask for is reported with that reason.
Where it runs
Section titled “Where it runs”A local run of b8e qa audit uses a browser on your machine, with your network. A cloud run, from
the CLI or the dashboard, uses a Binclusive-hosted VM that runs the journeys one after another. The
steps, outcomes and report are the same either way, and every run is stored on the project.