QUERY · ISSUE

Set threshold for Coveralls failure

openby dlechopened 2021-06-24updated 2025-11-06

proposed-close

Since MicroPython coverage tests have non-deterministic behavior, many lines of code are executed a different number of times in each CI run. Coveralls takes this number into account when calculating the coverage, so often we get "failures" due to variance between runs even though nothing actually changed. Coveralls provides a threshold setting for what constitutes failure. This could be set (or increased if it is already set) to reduce the number of nuisance "failures".

CANDIDATE · ISSUE

codecov does not run reports if any test fails

closedby dlechopened 2022-03-29updated 2022-11-29

tests

As documented in https://docs.codecov.com/docs/ci-service-relationship#section-checking-ci-status, codecov doesn't run reports if any CI test fails. However, it is quite common for the code size test to fail when adding new features and this is when code coverage reports are arguably the most useful.

I suggest that we change the code size test to never fail and make it always add a pull request comment with the report instead.

Assessment

Q Qwen · high

Both issues address CI/CD infrastructure problems related to code coverage reporting, but they concern different tools and different root causes. Issue #7455 is about Coveralls (a specific coverage service) and proposes setting a threshold to handle non-deterministic variance in coverage numbers. Issue #8464 is about Codecov (a different coverage service) and addresses the problem that Codecov skips reports entirely when any test fails, particularly the code size test. While both deal with coverage reporting reliability in CI, they propose different solutions for different services and different failure modes.

Suggested action

Keep both issues open as they address distinct problems. #7455 could be resolved by configuring Coveralls threshold settings, while #8464 requires changes to how the code size test behaves in the CI pipeline.

Set threshold for Coveralls failure

codecov does not run reports if any test fails

Keyboard