CI Metrics

Stout enables CI Metrics per organization, on request. To use CI Metrics, ask Stout to enable it for your organization.

CI Metrics shows what your hardware CI pipeline reports to Stout. It answers three questions: which merges ran on real hardware, which failures were real firmware bugs, and which tests are flaky. A repository can hold several projects, and each project can have its own hardware test suite. CI Metrics shows the metrics for one project, or for all projects together. The reports and the API name a project in the suite field. The configuration of the CI ingest tool lists the projects under products. Your CI sends two kinds of reports to Stout. It sends a merge announcement for each merge, and a run report for each hardware test run. Stout calculates the metrics from these reports. To tell real firmware bugs from bench problems, Stout also needs a description of the files in each project.

Set up reporting

Setup has five parts:

Create a CI key.
Send a merge announcement for each merge.
Send a run report for each hardware test run.
Describe the files of each project.
Report the status of each sync.

After parts 1 to 3, Stout shows merges, failures, and flaky tests. Part 4 lets Stout separate real firmware bugs from problems in the bench or the test harness. Part 5 lets the page show when the data was last brought up to date.

Create a CI key

A CI key lets your CI send reports to Stout. It works only for the report endpoints, and it cannot reach a box. The dashboard has no page for CI keys. Use the HTTP API with the access token of an owner or admin. See HTTP API for authentication.

curl -X POST https://api.stoutdata.ai/api/orgs/<org-id>/ci-keys \
  -H "Authorization: Bearer <access-token>" \
  -H "Content-Type: application/json" \
  -d '{"name": "firmware-ci"}'

The response contains the key in apiKey. The key starts with stout_, followed by 64 hexadecimal characters.

Before you close the response, store apiKey in the secret store of your CI. Stout shows the key only in this response. Stout cannot show the key again.

Action	Request	Who can do it
Create a key	`POST /api/orgs/<org-id>/ci-keys` with a `name` of 1 to 255 characters	Owners and admins
List keys	`GET /api/orgs/<org-id>/ci-keys`	Every member
Revoke a key	`DELETE /api/orgs/<org-id>/ci-keys/<key-id>`	Owners and admins

The list shows the name of each key and the times when someone created it, last used it, and revoked it. The list never shows the key itself. A revoked key stops working immediately.

Send a merge announcement

Send one merge announcement for each merge to the branch that you test. Send it from a CI job that does not need the hardware. Then Stout counts each merge, also when your hardware runner is offline. If the repository holds several projects, send one announcement for each project whose test suite the merge starts. A merge that changes code that two projects share gets two announcements.

curl -X POST https://api.stoutdata.ai/api/ci/merge-announce \
  -H "X-API-Key: <ci-key>" \
  -H "Content-Type: application/json" \
  -d '{"repo_key": "example-org/firmware", "suite": "sensor-board", "sha": "3f2a9c1", "branch": "main", "merged_at": "2026-09-01T17:04:00Z"}'

Field	Type	Required	Meaning
`repo_key`	String, 1 to 255 characters	Yes	A name for the repository. Use the same value in run reports.
`suite`	String, 1 to 255 characters	No	The project whose test suite the merge starts. If you leave it out, a run of any project can check the merge.
`sha`	String of 7 to 64 hexadecimal characters	Yes	The merge commit
`branch`	String, 1 to 255 characters	Yes	The branch that received the merge
`merged_at`	ISO 8601 date and time	Yes	The time of the merge

Stout returns HTTP 204. A second announcement with the same repo_key, suite, and sha changes nothing.

Send a run report

At the end of each attempt of a hardware test run, send one run report. Send it when the run passes and when it fails. If the run uses several bench boxes, send one report for each box.

curl -X POST https://api.stoutdata.ai/api/ci/run-report \
  -H "X-API-Key: <ci-key>" \
  -H "Content-Type: application/json" \
  -d @run-report.json

{
  "repo_key": "example-org/firmware",
  "workflow": "hardware-tests",
  "suite": "sensor-board",
  "run_id": 1042,
  "run_attempt": 1,
  "trigger": "merge",
  "branch": "main",
  "sha": "3f2a9c1",
  "started_at": "2026-09-01T17:06:00Z",
  "ended_at": "2026-09-01T17:19:00Z",
  "conclusion": "failure",
  "hello_ok": true,
  "box_key": "BENCH-1",
  "tests": [
    { "id": "test_power_rails", "outcome": "pass", "attempts": 1, "duration_s": 42.5 },
    { "id": "test_uart_boot", "outcome": "fail", "attempts": 3, "duration_s": 54.0, "verdict": "device" }
  ],
  "covers_shas": ["8b41d07", "3f2a9c1"],
  "changes": {
    "base_sha": "c90e2b4",
    "commits": [
      { "sha": "8b41d07", "subject": "Add a boot retry to the test harness", "files": ["tests/hil/boot.py"] },
      { "sha": "3f2a9c1", "subject": "Raise the UART baud rate", "files": ["firmware/uart.c"] }
    ]
  }
}

Field	Type	Required	Meaning
`repo_key`	String, 1 to 255 characters	Yes	The same value as in merge announcements
`workflow`	String, 1 to 255 characters	Yes	The name of the CI workflow
`suite`	String, 1 to 255 characters	No	The project that the test suite belongs to. If you leave it out, Stout uses the value of `workflow`.
`run_id`	Integer, 0 or more	Yes	The ID of the CI run
`run_attempt`	Integer, 1 or more	Yes	The attempt number of the CI run
`trigger`	`merge`, `nightly`, or `manual`	Yes	What started the run
`branch`	String, 1 to 255 characters	Yes	The branch that the run tested
`sha`	String of 7 to 64 hexadecimal characters	Yes	The commit that the run tested
`started_at`	ISO 8601 date and time	Yes	The start time of the run
`ended_at`	ISO 8601 date and time	Yes	The end time of the run
`conclusion`	String, 1 to 50 characters	Yes	The result of the run, as your CI reports it
`hello_ok`	Boolean	Yes	`false` if the bench stopped the run. For example, the connectivity check before the tests failed. Stout does not use the results of a run with `hello_ok` set to `false`. Otherwise `true`.
`box_key`	String, 1 to 255 characters	Yes	A name for the box or bench that ran the tests
`tests`	Array of up to 1,000 test items	Yes	One item for each test
`covers_shas`	Array of up to 1,000 commits, each 7 to 64 hexadecimal characters	No	Every commit in the push that started the run. A merge in this list counts as checked by this run.
`changes`	Object	No	The commits since the previous run of the same workflow on the same branch
`results_missing`	Boolean	No	`true` if the run happened, but its results are not available. For example, the results expired, or the run was cancelled. Stout then ignores `hello_ok` and `tests`. If you leave it out, Stout uses `false`.

Each item in tests has these fields:

Field	Type	Required	Meaning
`id`	String, 1 to 512 characters	Yes	A stable name for the test
`outcome`	`pass` or `fail`	Yes	`pass` if an attempt passed. `fail` if every attempt failed.
`attempts`	Integer, 1 or more	Yes	The number of attempts, retries included
`duration_s`	Number, 0 or more	Yes	The duration of the test in seconds
`verdict`	`device`, `infra`, or `unknown`	No	For a failed test: `device` if the test harness blamed the device under test, `infra` if it blamed the bench. If you leave it out, Stout uses `unknown`.

The changes object has these fields:

Field	Type	Required	Meaning
`base_sha`	String of 7 to 64 hexadecimal characters, or `null`	Yes	The commit that the previous run of the same workflow on the same branch tested. `null` if there is no previous run.
`commits`	Array of up to 250 commit items	Yes	The commits after `base_sha`, up to and including `sha`, oldest first. If the range has more than 250 commits, send the newest 250.
`range_shas`	Array of up to 5,000 commits, each 7 to 64 hexadecimal characters	No	Every commit after `base_sha`, up to and including `sha`, oldest first. Send it when the range has more commits than `commits` holds. If the range has more than 5,000 commits, send the newest 5,000.
`truncated`	Boolean	No	`true` if the range held more commits than you sent in `range_shas`. If you leave out `range_shas`, `true` if the range held more commits than you sent in `commits`.

Each item in commits has these fields:

Field	Type	Required	Meaning
`sha`	String of 7 to 64 hexadecimal characters	Yes	The commit
`subject`	String of up to 1,000 characters	Yes	The first line of the commit message
`committed_at`	ISO 8601 date and time	No	The time of the commit
`files`	Array of up to 2,000 paths	Yes	The files that the commit changed, relative to the root of the repository
`files_truncated`	Boolean	No	`true` if the commit changed more files than you sent

For a new attempt of the same run, send the same changes as for the first attempt.

The changes field sends the first line of each commit message and the paths of the changed files. It sends no source code. If a failed test has no verdict, or its run has no changes, Stout cannot decide whether the failure was a real firmware bug.

A box can pass the connectivity check and then fail before any test runs. For example, a new firmware build does not boot. Do not report this box with hello_ok set to false, because Stout then counts a firmware problem as a bench problem. Set hello_ok to true, and report one failed test with the id suite:bring-up and the verdict unknown. When the box runs its tests, report suite:bring-up as a pass. Then a later pass can resolve the failure. A test id that starts with suite: is not a test, so Stout does not show it in Failing tests. A run can happen without results that your CI can read. For example, the results expired before your CI read them, or a newer push cancelled the run. Report this run with results_missing set to true, an empty tests list, and its changes. Stout then uses the run only to find the merges that a later run tested. Stout returns HTTP 204. A second report with the same repo_key, run_id, run_attempt, and box_key replaces the first report. A report with results_missing set to true does not replace a report with results. If a request body breaks these rules, Stout returns HTTP 400. If the X-API-Key header has no key, a wrong key, or a revoked key, Stout returns HTTP 401.

Describe the files of each project

Stout finds real firmware bugs from the files that each commit changed. To do this, Stout must know which files are the firmware of a project, and which files are its test harness. Send one description for each project. When the files of a project move to other directories, send the description again.

curl -X PUT https://api.stoutdata.ai/api/ci/suite-config \
  -H "X-API-Key: <ci-key>" \
  -H "Content-Type: application/json" \
  -d @suite-config.json

{
  "repo_key": "example-org/firmware",
  "suite": "sensor-board",
  "firmware_paths": ["firmware/**", "!firmware/docs/**"],
  "harness_paths": ["tests/hil/**", ".github/workflows/sensor-board.yml"],
  "trigger_paths": ["firmware/**", "tests/hil/**"],
  "merge_trigger": true
}

Field	Type	Required	Meaning
`repo_key`	String, 1 to 255 characters	Yes	The same value as in the reports
`suite`	String, 1 to 255 characters	Yes	The project. Use the same value as in the reports.
`firmware_paths`	Array of up to 200 path patterns	Yes	The files that make up the firmware of the project
`harness_paths`	Array of up to 200 path patterns	Yes	The files that make up the test harness and the CI configuration of the project
`trigger_paths`	Array of up to 200 path patterns	No	The files that start the test suite of the project when a merge changes them
`merge_trigger`	Boolean	Yes	`false` if the test suite of the project never runs on a merge
`branches`	Array of 1 to 50 branch names	No	The branches whose runs count in the metrics of the project. `["*"]` counts every branch.

A path pattern is relative to the root of the repository. The patterns follow the paths filter of GitHub Actions, with fewer special characters:

* matches any characters in one directory level.
** matches any characters across directory levels.
A ! at the start excludes the files that the rest of the pattern matches. Put every ! pattern after all the other patterns.
A pattern that contains ?, +, [, ], {, }, or \ is not valid.

Stout returns HTTP 204. A new description replaces the old one. The next time that you open the page, Stout applies it to all earlier failures. Use branches to keep work in progress out of the metrics. For example, a person can start a run of a feature branch by hand. If branches is ["main"], that run does not count in any metric. Stout keeps the run, so a later change to branches applies to all earlier runs. If you never send branches, every branch counts. A description without branches keeps the branches that you sent before.

Report the status of each sync

If your CI sends its reports in batches, send a status at the end of each batch. Then the page shows when the data was last brought up to date. The page also shows a warning when the batches stop, or when a batch fails.

curl -X POST https://api.stoutdata.ai/api/ci/ingest-status \
  -H "X-API-Key: <ci-key>" \
  -H "Content-Type: application/json" \
  -d '{"repo_key": "example-org/firmware", "status": "ok", "started_at": "2026-09-01T18:00:00Z", "finished_at": "2026-09-01T18:02:10Z", "products": ["sensor-board"], "run_reports": 12, "merge_announcements": 3, "skipped_runs": 1, "warnings": 0}'

Field	Type	Required	Meaning
`repo_key`	String, 1 to 255 characters	Yes	The same value as in the reports
`status`	`ok` or `failed`	Yes	`ok` if the batch sent all of its reports. `failed` if the batch stopped.
`started_at`	ISO 8601 date and time	Yes	The start time of the batch
`finished_at`	ISO 8601 date and time	Yes	The end time of the batch
`products`	Array of up to 100 project names	Yes	The projects that the batch read
`run_reports`	Integer, 0 or more	Yes	The number of run reports that the batch sent
`merge_announcements`	Integer, 0 or more	Yes	The number of merge announcements that the batch sent
`skipped_runs`	Integer, 0 or more	Yes	The number of runs that the batch did not report
`warnings`	Integer, 0 or more	Yes	The number of warnings in the batch
`error`	String of up to 500 characters	No	Why the batch failed. Do not put a token or a key in it.
`tool_version`	String, 1 to 64 characters	No	The version of the program that sent the batch

Stout returns HTTP 204. Stout keeps one status for each repo_key. A failed batch does not change the time of the last successful batch. Stout ignores a status that ends before the status that it already has.

Read the metrics

To change the period, use the date control at the top of the page. 7d, 30d, and 90d select the last 7, 30, or 90 days. The page starts with the last 30 days. To see one project, select it in the project list. The list shows All projects until you select a project. In All projects, a merge that your CI announced for two projects counts two times. A merge that your CI announced without a project shows as Unassigned in All projects. The view of a single project does not count it.

Tile	What it counts
Merges checked on hardware	The announced merges in the period that a run checked. The tile also shows how many of them a later run checked.
Real FW bugs caught	The resolved failures that were real firmware bugs. The tile also shows the number of bench or harness problems, and the number of ambiguous failures.
Flaky failures	The failures on a commit where the same test also passed on the same box
Infra runs	The run reports with `hello_ok` set to `false`, out of all run reports in the period, except the runs without results. The bench stopped these runs, so Stout does not use their results. If your organization collects run records, this tile also counts the runs that the box stopped.

Each tile has an information icon (ⓘ). Click the icon to see what the tile counts and how to use it. Under the tiles, three tabs show the details. The page opens on the first tab. Each tab shows the number of its rows, and has its own information icon.

Tab	Contents
Failing tests	For each test that failed at least once: the number of boxes, run reports, failed attempts, and passes on a retry, the failure rate, and the flake rate. Each rate shows the counts that it comes from. The tests with the most passes on a retry come first, then the tests with the most failed attempts. A line under the table gives the number of the other tests.
Real FW bugs caught	The first failure of each real firmware bug, with the commit that broke the test and the commit that fixed it. Under it, Resolved by a bench or harness change shows the other resolved failures, each with the reason that it is not a firmware bug. That list is closed until you click its title.
Merges not checked on hardware	The announced merges that no run checked. In All projects, the table also shows the project.

When you select a different project or period, the page keeps the tab that you selected. A run checks a merge when the run reached the hardware and recorded at least one test. A suite:bring-up item is not a test. The run must also test one of these commits:

The merge commit.
A push that included the merge.
A later commit on the same branch, after a run that contained the merge.

The last case covers a merge whose own run did not finish. For example, a newer push can cancel the run of an older push. The next run tests a tree that contains the older merge, so the older merge counts as checked by a later run. Each project counts only the runs on its branches. Runs on other branches do not count in any metric.

Where the data starts

Your CI can keep test results for a limited time. If results expired before Stout read them, the data of a project starts later than the period that you selected. Then the page shows the date where the data of the project starts. A merge that landed before the first run of its project, and that no run checked, does not count in Merges checked on hardware. No data exists that can check it. The line that shows where the data starts also shows how many merges Stout left out.

Runs without results

A run report with results_missing set to true does not count in any metric, Infra runs included. If some runs in the period have no results, a line under the tiles shows how many. A run without results still tells Stout which commits the run contained. Thus a later run with results can check a merge that only the run without results tested.

Sync status

Next to the project list, the page shows when your CI last brought the data up to date. If your CI does not report the status of its syncs, the page shows Last synced: unknown. The page shows a warning in these conditions:

The last successful sync is older than 3 hours.
The last sync failed.

Owners and admins also see why the sync failed. The administrator of a self-hosted installation can change the limit of 3 hours.

How Stout links a run to its run records

A run record is the signed record that a box makes of each lager python run. If your organization collects run records, CI Metrics compares each run report with these records. If your organization does not collect run records, the page does not show this data. Stout links a run report to a run record when all of these conditions are true:

The record is from the same organization.
A test runner started the run on the box, not a person.
The run is not a run that Stout itself started on the box.
The Lager CLI sent the code from a git tree that has all its changes in commits.
The commit of the record is the full sha of the run report.
The run started on the box in the time of the run report, with 10 minutes of tolerance before and after.

When your organization collects run records, the page shows this data:

Merges checked on hardware shows how many checked merges have a run record. The run that checked the merge must have a linked record.
Infra runs also counts the runs that reached the bench, but the box stopped them. For these runs, each linked record says that the box disconnected, lost the run, or did not start it. A second line on the tile shows how many of the infra runs are these runs. A run that has a different exit on one or more of its records is not an infra run.
A line under the tiles shows how many runs have run records and how many do not. In All projects, the line also shows the number of test runner records that no run report links to.
In Real FW bugs caught, a commit that has a linked record has a Records link. The link opens the run records of that commit. A member sees only the records of the boxes that the member has access to.

Know these limits:

A run record gives the exit code of the script. An exit code of 0 does not mean that the tests passed.
A run report with a short sha does not link to a record.
The Lager CLI reports the commit. The box does not examine the code to find the commit.
The boxes must have Lager 0.56.0 or a later version to make run records.

How Stout labels a failure

A failure is a test with the outcome fail in a run that reached the hardware. Stout gives each failure one of three labels. Stout looks at each bench box on its own, so a pass on one box does not change a failure on another box.

Label	Rule
Flake	The same test passed on the same commit and on the same box, in any run report, at any time.
Resolved	The failure is not a flake, and the same test passed on a later commit on the same box and on the same branch.
Pending	Stout has no pass for the test that decides the label yet.

Stout calculates the labels each time that you open the page. A new report can change the label of an older failure. A test can fail on several commits in a row, with no pass between the failures. Stout counts these failures as one resolved failure. A run report with hello_ok set to false does not count in any metric, except Infra runs. If your organization collects run records, this rule also applies to a run that the box stopped. Refer to How Stout links a run to its run records. A run report with results_missing set to true does not count in any metric. Stout shows two rates for each test:

The failure rate is the number of failed attempts, divided by all attempts, as a percentage. A test that passed on its third attempt adds two failed attempts.
The flake rate is the number of passes on a retry, divided by the number of failed attempts, as a percentage.

If your CI never retries a test, the test has no flake rate, and the page shows a dash (—). A failure without a retry can be a real bug or a flaky test. Only a retry tells the two apart. A pass on a later commit of a different branch does not resolve a failure. A pass on the same commit makes a failure a flake on any branch. Each metric uses its own date:

Merges use merged_at.
Resolved failures use the start time of the first run that failed.
Flake rates and infra runs use the start time of each run.

How Stout finds real firmware bugs

Stout checks each resolved failure against three conditions. Stout counts the failure as a real firmware bug only if all three conditions are true:

The test harness blamed the device. The failed test has the verdict device. If your organization collects run records, the bench must also be the same. Look at a bench box that has run records on the failing commit and on the fix. The records must show the same nets and the same Lager version. They must also show the same instruments with the same firmware.
The commit under test changed firmware. The changes of the first run on the failing commit include a firmware file.
The fix changed firmware, and not the test harness. Between the failure and the fix, a commit changed a firmware file. No commit in that range changed only test harness files.

Stout checks the conditions in this order, and the first condition that fails gives the reason. If Stout has too little data to decide, it marks the failure Ambiguous. Real FW bugs caught names the firmware commit that broke the test and the commit that fixed it. If more than one commit qualifies, it shows the range of commits. Resolved by a bench or harness change shows every other resolved failure, with one of these reasons:

Reason on the page	Label	Meaning
Harness verdict: bench or infrastructure	Bench or harness	The test harness blamed the bench, not the device.
No harness verdict was reported	Ambiguous	The run report has no `verdict` for the failed test.
Bench changed between failure and fix (nets, Lager version or instrument firmware)	Ambiguous	The run records of the fix show a bench that the run records of the failure do not show, on the same box. The change to the bench can be the real fix. Stout uses this reason only if your organization collects run records.
No firmware and harness paths are configured for this project	Ambiguous	Stout has no description of the files of the project.
The changes under test are unknown	Ambiguous	The first run on the failing commit has no `changes`, or its `base_sha` is `null`.
The files changed under test were not fully reported	Ambiguous	No firmware file shows in the changes under test, and some commits or files were not sent.
Commit under test did not touch firmware	Bench or harness	No commit under test changed a firmware file.
The changes before the fix are not fully recorded	Ambiguous	The run reports do not connect the failing commit to the fix without a gap.
Fix touched harness paths	Bench or harness	A commit between the failure and the fix changed test harness files and no firmware files.
Fix did not touch firmware	Bench or harness	No commit between the failure and the fix changed a firmware file.
Fix changed firmware and harness in the same commit	Ambiguous	One commit changed both kinds of files, so the fix is not clear.
The files changed before the fix were not fully reported	Ambiguous	Some commits before the fix have files that were not sent.

Confirm or dismiss a failure

Each person who can see CI Metrics can correct the automatic label of a resolved failure. A review replaces the automatic label. Stout keeps the review when it calculates the labels again.

To count a failure from Resolved by a bench or harness change as a real firmware bug, click Confirm on its row.
To remove a failure from Real FW bugs caught, click Dismiss on its row. Stout moves the failure to the other list, with the reason Dismissed by a reviewer.
To go back to the automatic label, click Reset on the row.

A reviewed row shows who confirmed or dismissed the failure. A row without a review shows nothing. The audit log records each review, with the person who made it.

Boxes

Workbench

Factory

Organization & Settings

CI Metrics

Set up reporting

Create a CI key

Send a merge announcement

Send a run report

Describe the files of each project

Report the status of each sync

Read the metrics

Where the data starts

Runs without results

Sync status

How Stout links a run to its run records

How Stout labels a failure

How Stout finds real firmware bugs

Confirm or dismiss a failure

​Set up reporting

​Create a CI key

​Send a merge announcement

​Send a run report

​Describe the files of each project

​Report the status of each sync

​Read the metrics

​Where the data starts

​Runs without results

​Sync status

​How Stout links a run to its run records

​How Stout labels a failure

​How Stout finds real firmware bugs

​Confirm or dismiss a failure

Set up reporting

Create a CI key

Send a merge announcement

Send a run report

Describe the files of each project

Report the status of each sync

Read the metrics

Where the data starts

Runs without results

Sync status

How Stout links a run to its run records

How Stout labels a failure

How Stout finds real firmware bugs

Confirm or dismiss a failure