CI Metrics

Stout enables CI Metrics per organization, on request. To use CI Metrics, ask Stout to enable it for your organization.

CI Metrics shows what your hardware CI pipeline reports to Stout. It answers three questions: which merges ran on real hardware, which failures were real firmware bugs, and which tests are flaky. A repository can hold several products, and each product can have its own hardware test suite. CI Metrics shows the metrics for one product, or for all products together. Your CI sends two kinds of reports to Stout. It sends a merge announcement for each merge, and a run report for each hardware test run. Stout calculates the metrics from these reports. To tell real firmware bugs from bench problems, Stout also needs a description of the files in each product.

Set up reporting

Setup has four parts:

Create a CI key.
Send a merge announcement for each merge.
Send a run report for each hardware test run.
Describe the files of each product.

After parts 1 to 3, Stout shows merges, failures, and flaky tests. Part 4 lets Stout separate real firmware bugs from problems in the bench or the test harness.

Create a CI key

A CI key lets your CI send reports to Stout. It works only for the report endpoints, and it cannot reach a box. The dashboard has no page for CI keys. Use the HTTP API with the access token of an owner or admin. See HTTP API for authentication.

curl -X POST https://api.stoutdata.ai/api/orgs/<org-id>/ci-keys \
  -H "Authorization: Bearer <access-token>" \
  -H "Content-Type: application/json" \
  -d '{"name": "firmware-ci"}'

The response contains the key in apiKey. The key starts with stout_, followed by 64 hexadecimal characters.

Before you close the response, store apiKey in the secret store of your CI. Stout shows the key only in this response. Stout cannot show the key again.

Action	Request	Who can do it
Create a key	`POST /api/orgs/<org-id>/ci-keys` with a `name` of 1 to 255 characters	Owners and admins
List keys	`GET /api/orgs/<org-id>/ci-keys`	Every member
Revoke a key	`DELETE /api/orgs/<org-id>/ci-keys/<key-id>`	Owners and admins

The list shows the name of each key and the times when someone created it, last used it, and revoked it. The list never shows the key itself. A revoked key stops working immediately.

Send a merge announcement

Send one merge announcement for each merge to the branch that you test. Send it from a CI job that does not need the hardware. Then Stout counts each merge, also when your hardware runner is offline. If the repository holds several products, send one announcement for each product whose test suite the merge starts. A merge that changes code that two products share gets two announcements.

curl -X POST https://api.stoutdata.ai/api/ci/merge-announce \
  -H "X-API-Key: <ci-key>" \
  -H "Content-Type: application/json" \
  -d '{"repo_key": "example-org/firmware", "suite": "sensor-board", "sha": "3f2a9c1", "branch": "main", "merged_at": "2026-09-01T17:04:00Z"}'

Field	Type	Required	Meaning
`repo_key`	String, 1 to 255 characters	Yes	A name for the repository. Use the same value in run reports.
`suite`	String, 1 to 255 characters	No	The product whose test suite the merge starts. If you leave it out, a run of any product can check the merge.
`sha`	String of 7 to 64 hexadecimal characters	Yes	The merge commit
`branch`	String, 1 to 255 characters	Yes	The branch that received the merge
`merged_at`	ISO 8601 date and time	Yes	The time of the merge

Stout returns HTTP 204. A second announcement with the same repo_key, suite, and sha changes nothing.

Send a run report

At the end of each attempt of a hardware test run, send one run report. Send it when the run passes and when it fails. If the run uses several bench boxes, send one report for each box.

curl -X POST https://api.stoutdata.ai/api/ci/run-report \
  -H "X-API-Key: <ci-key>" \
  -H "Content-Type: application/json" \
  -d @run-report.json

{
  "repo_key": "example-org/firmware",
  "workflow": "hardware-tests",
  "suite": "sensor-board",
  "run_id": 1042,
  "run_attempt": 1,
  "trigger": "merge",
  "branch": "main",
  "sha": "3f2a9c1",
  "started_at": "2026-09-01T17:06:00Z",
  "ended_at": "2026-09-01T17:19:00Z",
  "conclusion": "failure",
  "hello_ok": true,
  "box_key": "BENCH-1",
  "tests": [
    { "id": "test_power_rails", "outcome": "pass", "attempts": 1, "duration_s": 42.5 },
    { "id": "test_uart_boot", "outcome": "fail", "attempts": 3, "duration_s": 54.0, "verdict": "device" }
  ],
  "covers_shas": ["8b41d07", "3f2a9c1"],
  "changes": {
    "base_sha": "c90e2b4",
    "commits": [
      { "sha": "8b41d07", "subject": "Add a boot retry to the test harness", "files": ["tests/hil/boot.py"] },
      { "sha": "3f2a9c1", "subject": "Raise the UART baud rate", "files": ["firmware/uart.c"] }
    ]
  }
}

Field	Type	Required	Meaning
`repo_key`	String, 1 to 255 characters	Yes	The same value as in merge announcements
`workflow`	String, 1 to 255 characters	Yes	The name of the CI workflow
`suite`	String, 1 to 255 characters	No	The product that the test suite belongs to. If you leave it out, Stout uses the value of `workflow`.
`run_id`	Integer, 0 or more	Yes	The ID of the CI run
`run_attempt`	Integer, 1 or more	Yes	The attempt number of the CI run
`trigger`	`merge`, `nightly`, or `manual`	Yes	What started the run
`branch`	String, 1 to 255 characters	Yes	The branch that the run tested
`sha`	String of 7 to 64 hexadecimal characters	Yes	The commit that the run tested
`started_at`	ISO 8601 date and time	Yes	The start time of the run
`ended_at`	ISO 8601 date and time	Yes	The end time of the run
`conclusion`	String, 1 to 50 characters	Yes	The result of the run, as your CI reports it
`hello_ok`	Boolean	Yes	`true` if the run reached the hardware. `false` if the connectivity check before the tests failed.
`box_key`	String, 1 to 255 characters	Yes	A name for the box or bench that ran the tests
`tests`	Array of up to 1,000 test items	Yes	One item for each test
`covers_shas`	Array of up to 1,000 commits, each 7 to 64 hexadecimal characters	No	Every commit in the push that started the run. A merge in this list counts as checked by this run.
`changes`	Object	No	The commits since the previous run of the same workflow on the same branch

Each item in tests has these fields:

Field	Type	Required	Meaning
`id`	String, 1 to 512 characters	Yes	A stable name for the test
`outcome`	`pass` or `fail`	Yes	`pass` if an attempt passed. `fail` if every attempt failed.
`attempts`	Integer, 1 or more	Yes	The number of attempts, retries included
`duration_s`	Number, 0 or more	Yes	The duration of the test in seconds
`verdict`	`device`, `infra`, or `unknown`	No	For a failed test: `device` if the test harness blamed the device under test, `infra` if it blamed the bench. If you leave it out, Stout uses `unknown`.

The changes object has these fields:

Field	Type	Required	Meaning
`base_sha`	String of 7 to 64 hexadecimal characters, or `null`	Yes	The commit that the previous run of the same workflow on the same branch tested. `null` if there is no previous run.
`commits`	Array of up to 250 commit items	Yes	The commits after `base_sha`, up to and including `sha`, oldest first
`truncated`	Boolean	No	`true` if the range held more commits than you sent

Each item in commits has these fields:

Field	Type	Required	Meaning
`sha`	String of 7 to 64 hexadecimal characters	Yes	The commit
`subject`	String of up to 1,000 characters	Yes	The first line of the commit message
`committed_at`	ISO 8601 date and time	No	The time of the commit
`files`	Array of up to 2,000 paths	Yes	The files that the commit changed, relative to the root of the repository
`files_truncated`	Boolean	No	`true` if the commit changed more files than you sent

For a new attempt of the same run, send the same changes as for the first attempt.

The changes field sends the first line of each commit message and the paths of the changed files. It sends no source code. If a failed test has no verdict, or its run has no changes, Stout cannot decide whether the failure was a real firmware bug.

Stout returns HTTP 204. A second report with the same repo_key, run_id, run_attempt, and box_key replaces the first report. If a request body breaks these rules, Stout returns HTTP 400. If the X-API-Key header has no key, a wrong key, or a revoked key, Stout returns HTTP 401.

Describe the files of each product

Stout finds real firmware bugs from the files that each commit changed. To do this, Stout must know which files are the firmware of a product, and which files are its test harness. Send one description for each product. When the files of a product move to other directories, send the description again.

curl -X PUT https://api.stoutdata.ai/api/ci/suite-config \
  -H "X-API-Key: <ci-key>" \
  -H "Content-Type: application/json" \
  -d @suite-config.json

{
  "repo_key": "example-org/firmware",
  "suite": "sensor-board",
  "firmware_paths": ["firmware/**", "!firmware/docs/**"],
  "harness_paths": ["tests/hil/**", ".github/workflows/sensor-board.yml"],
  "trigger_paths": ["firmware/**", "tests/hil/**"],
  "merge_trigger": true
}

Field	Type	Required	Meaning
`repo_key`	String, 1 to 255 characters	Yes	The same value as in the reports
`suite`	String, 1 to 255 characters	Yes	The product. Use the same value as in the reports.
`firmware_paths`	Array of up to 200 path patterns	Yes	The files that make up the firmware of the product
`harness_paths`	Array of up to 200 path patterns	Yes	The files that make up the test harness and the CI configuration of the product
`trigger_paths`	Array of up to 200 path patterns	No	The files that start the test suite of the product when a merge changes them
`merge_trigger`	Boolean	Yes	`false` if the test suite of the product never runs on a merge

A path pattern is relative to the root of the repository. The patterns follow the paths filter of GitHub Actions, with fewer special characters:

* matches any characters in one directory level.
** matches any characters across directory levels.
A ! at the start excludes the files that the rest of the pattern matches. Put every ! pattern after all the other patterns.
A pattern that contains ?, +, [, ], {, }, or \ is not valid.

Stout returns HTTP 204. A new description replaces the old one. The next time that you open the page, Stout applies it to all earlier failures.

Read the metrics

To change the period, use the date control at the top of the page. 7d, 30d, and 90d select the last 7, 30, or 90 days. The page starts with the last 30 days. To see one product, select it in the product list. The list shows All products until you select a product. In All products, a merge that your CI announced for two products counts two times. A merge that your CI announced without a product shows as Unassigned in All products. The view of a single product does not count it.

Tile	What it counts
Merges checked on hardware	The announced merges in the period that a run checked. A run checks a merge when it reached the hardware, recorded at least one test, and tested the merge commit or a push that included it.
Real FW bugs caught	The resolved failures that were real firmware bugs. The tile also shows the number of bench or harness problems, and the number of ambiguous failures.
Flaky failures	The failures on a commit where the same test also passed on the same box
Infra runs	The run reports with `hello_ok` set to `false`, out of all run reports in the period

Table	Contents
Merges not checked on hardware	The announced merges that no run checked. In All products, the table also shows the product.
Real FW bugs caught	The first failure of each real firmware bug, with the commit that broke the test, the commit that fixed it, and the verdict of the test harness
Resolved by a bench or harness change	The other resolved failures, each with the reason that it is not a firmware bug. The list is closed until you click its title.
Flakiest tests	For each test: the number of boxes, run reports, attempts, failed attempts, and passes on a retry, and the flake rate

Two merges can land in separate pushes a few seconds apart. Then the bench can test only the second push, and the first merge shows as not checked.

How Stout labels a failure

A failure is a test with the outcome fail in a run that reached the hardware. Stout gives each failure one of three labels. Stout looks at each bench box on its own, so a pass on one box does not change a failure on another box.

Label	Rule
Flake	The same test passed on the same commit and on the same box, in any run report, at any time.
Resolved	The failure is not a flake, and the same test passed on a later commit on the same box.
Pending	Stout has no pass for the test that decides the label yet.

Stout calculates the labels each time that you open the page. A new report can change the label of an older failure. A test can fail on several commits in a row, with no pass between the failures. Stout counts these failures as one resolved failure. A run report with hello_ok set to false does not count in any metric, except Infra runs. The flake rate is the number of failed attempts, divided by all attempts, as a percentage. A test that passed on its third attempt adds two failed attempts. Each metric uses its own date:

Merges use merged_at.
Resolved failures use the start time of the first run that failed.
Flake rates and infra runs use the start time of each run.

How Stout finds real firmware bugs

Stout checks each resolved failure against three conditions. Stout counts the failure as a real firmware bug only if all three conditions are true:

The test harness blamed the device. The failed test has the verdict device.
The commit under test changed firmware. The changes of the first run on the failing commit include a firmware file.
The fix changed firmware, and not the test harness. Between the failure and the fix, a commit changed a firmware file. No commit in that range changed only test harness files.

Stout checks the conditions in this order, and the first condition that fails gives the reason. If Stout has too little data to decide, it marks the failure Ambiguous. Real FW bugs caught names the firmware commit that broke the test and the commit that fixed it. If more than one commit qualifies, it shows the range of commits. Resolved by a bench or harness change shows every other resolved failure, with one of these reasons:

Reason on the page	Label	Meaning
Harness verdict: bench or infrastructure	Bench or harness	The test harness blamed the bench, not the device.
No harness verdict was reported	Ambiguous	The run report has no `verdict` for the failed test.
No firmware and harness paths are configured for this product	Ambiguous	Stout has no description of the files of the product.
The changes under test are unknown	Ambiguous	The first run on the failing commit has no `changes`, or its `base_sha` is `null`.
The files changed under test were not fully reported	Ambiguous	No firmware file shows in the changes under test, and some commits or files were not sent.
Commit under test did not touch firmware	Bench or harness	No commit under test changed a firmware file.
The changes before the fix are not fully recorded	Ambiguous	The run reports do not connect the failing commit to the fix without a gap.
Fix touched harness paths	Bench or harness	A commit between the failure and the fix changed test harness files and no firmware files.
Fix did not touch firmware	Bench or harness	No commit between the failure and the fix changed a firmware file.
Fix changed firmware and harness in the same commit	Ambiguous	One commit changed both kinds of files, so the fix is not clear.
The files changed before the fix were not fully reported	Ambiguous	Some commits before the fix have files that were not sent.

Confirm or dismiss a failure

Organization owners and admins can correct the automatic label of a resolved failure. A review replaces the automatic label. Stout keeps the review when it calculates the labels again.

To count a failure from Resolved by a bench or harness change as a real firmware bug, click Confirm on its row.
To remove a failure from Real FW bugs caught, click Dismiss on its row. Stout moves the failure to the other list, with the reason Dismissed by an admin.
To go back to the automatic label, click Reset on the row.

The Status column shows Automatic, or the person who confirmed or dismissed the failure. The audit log records each review.

Boxes

Workbench

Factory

Organization & Settings

CI Metrics

Set up reporting

Create a CI key

Send a merge announcement

Send a run report

Describe the files of each product

Read the metrics

How Stout labels a failure

How Stout finds real firmware bugs

Confirm or dismiss a failure

​Set up reporting

​Create a CI key

​Send a merge announcement

​Send a run report

​Describe the files of each product

​Read the metrics

​How Stout labels a failure

​How Stout finds real firmware bugs

​Confirm or dismiss a failure

Set up reporting

Create a CI key

Send a merge announcement

Send a run report

Describe the files of each product

Read the metrics

How Stout labels a failure

How Stout finds real firmware bugs

Confirm or dismiss a failure