> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stoutdata.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# CI Metrics
> See which merges your CI checked on real hardware, which failures were real firmware bugs, and which tests are flaky.
Stout enables CI Metrics per organization, on request. To use CI Metrics, ask Stout to enable it for your organization.
**CI Metrics** shows what your hardware CI pipeline reports to Stout. It answers three questions: which merges ran on real hardware, which failures were real firmware bugs, and which tests are flaky.
A repository can hold several products, and each product can have its own hardware test suite. CI Metrics shows the metrics for one product, or for all products together.
Your CI sends two kinds of reports to Stout. It sends a merge announcement for each merge, and a run report for each hardware test run. Stout calculates the metrics from these reports. To tell real firmware bugs from bench problems, Stout also needs a description of the files in each product.
## Set up reporting
Setup has four parts:
1. Create a CI key.
2. Send a merge announcement for each merge.
3. Send a run report for each hardware test run.
4. Describe the files of each product.
After parts 1 to 3, Stout shows merges, failures, and flaky tests. Part 4 lets Stout separate real firmware bugs from problems in the bench or the test harness.
### Create a CI key
A CI key lets your CI send reports to Stout. It works only for the report endpoints, and it cannot reach a box.
The dashboard has no page for CI keys. Use the HTTP API with the access token of an owner or admin. See [HTTP API](/source/api/overview) for authentication.
```bash theme={null}
curl -X POST https://api.stoutdata.ai/api/orgs//ci-keys \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"name": "firmware-ci"}'
```
The response contains the key in `apiKey`. The key starts with `stout_`, followed by 64 hexadecimal characters.
Before you close the response, store `apiKey` in the secret store of your CI. Stout shows the key only in this response. Stout cannot show the key again.
| Action | Request | Who can do it |
| ------------ | ---------------------------------------------------------------------- | ----------------- |
| Create a key | `POST /api/orgs//ci-keys` with a `name` of 1 to 255 characters | Owners and admins |
| List keys | `GET /api/orgs//ci-keys` | Every member |
| Revoke a key | `DELETE /api/orgs//ci-keys/` | Owners and admins |
The list shows the name of each key and the times when someone created it, last used it, and revoked it. The list never shows the key itself. A revoked key stops working immediately.
### Send a merge announcement
Send one merge announcement for each merge to the branch that you test. Send it from a CI job that does not need the hardware. Then Stout counts each merge, also when your hardware runner is offline.
If the repository holds several products, send one announcement for each product whose test suite the merge starts. A merge that changes code that two products share gets two announcements.
```bash theme={null}
curl -X POST https://api.stoutdata.ai/api/ci/merge-announce \
-H "X-API-Key: " \
-H "Content-Type: application/json" \
-d '{"repo_key": "example-org/firmware", "suite": "sensor-board", "sha": "3f2a9c1", "branch": "main", "merged_at": "2026-09-01T17:04:00Z"}'
```
| Field | Type | Required | Meaning |
| ----------- | ---------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------- |
| `repo_key` | String, 1 to 255 characters | Yes | A name for the repository. Use the same value in run reports. |
| `suite` | String, 1 to 255 characters | No | The product whose test suite the merge starts. If you leave it out, a run of any product can check the merge. |
| `sha` | String of 7 to 64 hexadecimal characters | Yes | The merge commit |
| `branch` | String, 1 to 255 characters | Yes | The branch that received the merge |
| `merged_at` | ISO 8601 date and time | Yes | The time of the merge |
Stout returns HTTP 204. A second announcement with the same `repo_key`, `suite`, and `sha` changes nothing.
### Send a run report
At the end of each attempt of a hardware test run, send one run report. Send it when the run passes and when it fails. If the run uses several bench boxes, send one report for each box.
```bash theme={null}
curl -X POST https://api.stoutdata.ai/api/ci/run-report \
-H "X-API-Key: " \
-H "Content-Type: application/json" \
-d @run-report.json
```
```json theme={null}
{
"repo_key": "example-org/firmware",
"workflow": "hardware-tests",
"suite": "sensor-board",
"run_id": 1042,
"run_attempt": 1,
"trigger": "merge",
"branch": "main",
"sha": "3f2a9c1",
"started_at": "2026-09-01T17:06:00Z",
"ended_at": "2026-09-01T17:19:00Z",
"conclusion": "failure",
"hello_ok": true,
"box_key": "BENCH-1",
"tests": [
{ "id": "test_power_rails", "outcome": "pass", "attempts": 1, "duration_s": 42.5 },
{ "id": "test_uart_boot", "outcome": "fail", "attempts": 3, "duration_s": 54.0, "verdict": "device" }
],
"covers_shas": ["8b41d07", "3f2a9c1"],
"changes": {
"base_sha": "c90e2b4",
"commits": [
{ "sha": "8b41d07", "subject": "Add a boot retry to the test harness", "files": ["tests/hil/boot.py"] },
{ "sha": "3f2a9c1", "subject": "Raise the UART baud rate", "files": ["firmware/uart.c"] }
]
}
}
```
| Field | Type | Required | Meaning |
| ------------- | ----------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------- |
| `repo_key` | String, 1 to 255 characters | Yes | The same value as in merge announcements |
| `workflow` | String, 1 to 255 characters | Yes | The name of the CI workflow |
| `suite` | String, 1 to 255 characters | No | The product that the test suite belongs to. If you leave it out, Stout uses the value of `workflow`. |
| `run_id` | Integer, 0 or more | Yes | The ID of the CI run |
| `run_attempt` | Integer, 1 or more | Yes | The attempt number of the CI run |
| `trigger` | `merge`, `nightly`, or `manual` | Yes | What started the run |
| `branch` | String, 1 to 255 characters | Yes | The branch that the run tested |
| `sha` | String of 7 to 64 hexadecimal characters | Yes | The commit that the run tested |
| `started_at` | ISO 8601 date and time | Yes | The start time of the run |
| `ended_at` | ISO 8601 date and time | Yes | The end time of the run |
| `conclusion` | String, 1 to 50 characters | Yes | The result of the run, as your CI reports it |
| `hello_ok` | Boolean | Yes | `true` if the run reached the hardware. `false` if the connectivity check before the tests failed. |
| `box_key` | String, 1 to 255 characters | Yes | A name for the box or bench that ran the tests |
| `tests` | Array of up to 1,000 test items | Yes | One item for each test |
| `covers_shas` | Array of up to 1,000 commits, each 7 to 64 hexadecimal characters | No | Every commit in the push that started the run. A merge in this list counts as checked by this run. |
| `changes` | Object | No | The commits since the previous run of the same workflow on the same branch |
Each item in `tests` has these fields:
| Field | Type | Required | Meaning |
| ------------ | ------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | String, 1 to 512 characters | Yes | A stable name for the test |
| `outcome` | `pass` or `fail` | Yes | `pass` if an attempt passed. `fail` if every attempt failed. |
| `attempts` | Integer, 1 or more | Yes | The number of attempts, retries included |
| `duration_s` | Number, 0 or more | Yes | The duration of the test in seconds |
| `verdict` | `device`, `infra`, or `unknown` | No | For a failed test: `device` if the test harness blamed the device under test, `infra` if it blamed the bench. If you leave it out, Stout uses `unknown`. |
The `changes` object has these fields:
| Field | Type | Required | Meaning |
| ----------- | --------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
| `base_sha` | String of 7 to 64 hexadecimal characters, or `null` | Yes | The commit that the previous run of the same workflow on the same branch tested. `null` if there is no previous run. |
| `commits` | Array of up to 250 commit items | Yes | The commits after `base_sha`, up to and including `sha`, oldest first |
| `truncated` | Boolean | No | `true` if the range held more commits than you sent |
Each item in `commits` has these fields:
| Field | Type | Required | Meaning |
| ----------------- | ---------------------------------------- | -------- | ------------------------------------------------------------------------- |
| `sha` | String of 7 to 64 hexadecimal characters | Yes | The commit |
| `subject` | String of up to 1,000 characters | Yes | The first line of the commit message |
| `committed_at` | ISO 8601 date and time | No | The time of the commit |
| `files` | Array of up to 2,000 paths | Yes | The files that the commit changed, relative to the root of the repository |
| `files_truncated` | Boolean | No | `true` if the commit changed more files than you sent |
For a new attempt of the same run, send the same `changes` as for the first attempt.
The `changes` field sends the first line of each commit message and the paths of the changed files. It sends no source code. If a failed test has no `verdict`, or its run has no `changes`, Stout cannot decide whether the failure was a real firmware bug.
Stout returns HTTP 204. A second report with the same `repo_key`, `run_id`, `run_attempt`, and `box_key` replaces the first report.
If a request body breaks these rules, Stout returns HTTP 400. If the `X-API-Key` header has no key, a wrong key, or a revoked key, Stout returns HTTP 401.
### Describe the files of each product
Stout finds real firmware bugs from the files that each commit changed. To do this, Stout must know which files are the firmware of a product, and which files are its test harness.
Send one description for each product. When the files of a product move to other directories, send the description again.
```bash theme={null}
curl -X PUT https://api.stoutdata.ai/api/ci/suite-config \
-H "X-API-Key: " \
-H "Content-Type: application/json" \
-d @suite-config.json
```
```json theme={null}
{
"repo_key": "example-org/firmware",
"suite": "sensor-board",
"firmware_paths": ["firmware/**", "!firmware/docs/**"],
"harness_paths": ["tests/hil/**", ".github/workflows/sensor-board.yml"],
"trigger_paths": ["firmware/**", "tests/hil/**"],
"merge_trigger": true
}
```
| Field | Type | Required | Meaning |
| ---------------- | -------------------------------- | -------- | ------------------------------------------------------------------------------- |
| `repo_key` | String, 1 to 255 characters | Yes | The same value as in the reports |
| `suite` | String, 1 to 255 characters | Yes | The product. Use the same value as in the reports. |
| `firmware_paths` | Array of up to 200 path patterns | Yes | The files that make up the firmware of the product |
| `harness_paths` | Array of up to 200 path patterns | Yes | The files that make up the test harness and the CI configuration of the product |
| `trigger_paths` | Array of up to 200 path patterns | No | The files that start the test suite of the product when a merge changes them |
| `merge_trigger` | Boolean | Yes | `false` if the test suite of the product never runs on a merge |
A path pattern is relative to the root of the repository. The patterns follow the `paths` filter of GitHub Actions, with fewer special characters:
* `*` matches any characters in one directory level.
* `**` matches any characters across directory levels.
* A `!` at the start excludes the files that the rest of the pattern matches. Put every `!` pattern after all the other patterns.
* A pattern that contains `?`, `+`, `[`, `]`, `{`, `}`, or `\` is not valid.
Stout returns HTTP 204. A new description replaces the old one. The next time that you open the page, Stout applies it to all earlier failures.
## Read the metrics
To change the period, use the date control at the top of the page. **7d**, **30d**, and **90d** select the last 7, 30, or 90 days. The page starts with the last 30 days.
To see one product, select it in the product list. The list shows **All products** until you select a product. In **All products**, a merge that your CI announced for two products counts two times.
A merge that your CI announced without a product shows as **Unassigned** in **All products**. The view of a single product does not count it.
| Tile | What it counts |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Merges checked on hardware** | The announced merges in the period that a run checked. A run checks a merge when it reached the hardware, recorded at least one test, and tested the merge commit or a push that included it. |
| **Real FW bugs caught** | The resolved failures that were real firmware bugs. The tile also shows the number of bench or harness problems, and the number of ambiguous failures. |
| **Flaky failures** | The failures on a commit where the same test also passed on the same box |
| **Infra runs** | The run reports with `hello_ok` set to `false`, out of all run reports in the period |
| Table | Contents |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **Merges not checked on hardware** | The announced merges that no run checked. In **All products**, the table also shows the product. |
| **Real FW bugs caught** | The first failure of each real firmware bug, with the commit that broke the test, the commit that fixed it, and the verdict of the test harness |
| **Resolved by a bench or harness change** | The other resolved failures, each with the reason that it is not a firmware bug. The list is closed until you click its title. |
| **Flakiest tests** | For each test: the number of boxes, run reports, attempts, failed attempts, and passes on a retry, and the flake rate |
Two merges can land in separate pushes a few seconds apart. Then the bench can test only the second push, and the first merge shows as not checked.
### How Stout labels a failure
A failure is a test with the outcome `fail` in a run that reached the hardware. Stout gives each failure one of three labels. Stout looks at each bench box on its own, so a pass on one box does not change a failure on another box.
| Label | Rule |
| -------- | -------------------------------------------------------------------------------------------- |
| Flake | The same test passed on the same commit and on the same box, in any run report, at any time. |
| Resolved | The failure is not a flake, and the same test passed on a later commit on the same box. |
| Pending | Stout has no pass for the test that decides the label yet. |
Stout calculates the labels each time that you open the page. A new report can change the label of an older failure.
A test can fail on several commits in a row, with no pass between the failures. Stout counts these failures as one resolved failure.
A run report with `hello_ok` set to `false` does not count in any metric, except **Infra runs**.
The flake rate is the number of failed attempts, divided by all attempts, as a percentage. A test that passed on its third attempt adds two failed attempts.
Each metric uses its own date:
* Merges use `merged_at`.
* Resolved failures use the start time of the first run that failed.
* Flake rates and infra runs use the start time of each run.
### How Stout finds real firmware bugs
Stout checks each resolved failure against three conditions. Stout counts the failure as a real firmware bug only if all three conditions are true:
1. **The test harness blamed the device.** The failed test has the verdict `device`.
2. **The commit under test changed firmware.** The `changes` of the first run on the failing commit include a firmware file.
3. **The fix changed firmware, and not the test harness.** Between the failure and the fix, a commit changed a firmware file. No commit in that range changed only test harness files.
Stout checks the conditions in this order, and the first condition that fails gives the reason. If Stout has too little data to decide, it marks the failure **Ambiguous**.
**Real FW bugs caught** names the firmware commit that broke the test and the commit that fixed it. If more than one commit qualifies, it shows the range of commits.
**Resolved by a bench or harness change** shows every other resolved failure, with one of these reasons:
| Reason on the page | Label | Meaning |
| ------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------ |
| Harness verdict: bench or infrastructure | Bench or harness | The test harness blamed the bench, not the device. |
| No harness verdict was reported | Ambiguous | The run report has no `verdict` for the failed test. |
| No firmware and harness paths are configured for this product | Ambiguous | Stout has no description of the files of the product. |
| The changes under test are unknown | Ambiguous | The first run on the failing commit has no `changes`, or its `base_sha` is `null`. |
| The files changed under test were not fully reported | Ambiguous | No firmware file shows in the changes under test, and some commits or files were not sent. |
| Commit under test did not touch firmware | Bench or harness | No commit under test changed a firmware file. |
| The changes before the fix are not fully recorded | Ambiguous | The run reports do not connect the failing commit to the fix without a gap. |
| Fix touched harness paths | Bench or harness | A commit between the failure and the fix changed test harness files and no firmware files. |
| Fix did not touch firmware | Bench or harness | No commit between the failure and the fix changed a firmware file. |
| Fix changed firmware and harness in the same commit | Ambiguous | One commit changed both kinds of files, so the fix is not clear. |
| The files changed before the fix were not fully reported | Ambiguous | Some commits before the fix have files that were not sent. |
### Confirm or dismiss a failure
Organization owners and admins can correct the automatic label of a resolved failure. A review replaces the automatic label. Stout keeps the review when it calculates the labels again.
* To count a failure from **Resolved by a bench or harness change** as a real firmware bug, click **Confirm** on its row.
* To remove a failure from **Real FW bugs caught**, click **Dismiss** on its row. Stout moves the failure to the other list, with the reason **Dismissed by an admin**.
* To go back to the automatic label, click **Reset** on the row.
The **Status** column shows **Automatic**, or the person who confirmed or dismissed the failure. The audit log records each review.