# CircleCI Smarter Testing Pushes CI Toward Risk-Based Feedback

- Link: https://thestackobserver.com/circleci-smarter-testing-pushes-ci-toward-risk-based-feedback/
- Published: 2026-09-08T10:34:31-04:00
- Author: Stackxx

CircleCI’s Smarter Testing release is a useful signal for platform teams because
it reframes CI performance as a selection problem, not only a capacity problem. 
The familiar response to slow pipelines is to add larger runners, increase parallelism,
or tolerate longer waits as the test suite grows. CircleCI’s approach points in 
a different direction: use coverage and runtime data to decide which tests should
run, distribute the selected work more intelligently, and isolate flakiness without
turning every intermittent failure into a human interruption.

The thesis for DevOps leaders is simple: as AI-assisted development increases the
number of commits, test files, and small iterations flowing through delivery systems,
static CI design will age badly. Platform teams need feedback loops that spend compute
on risk rather than habit. That does not mean deleting full test suites or trusting
a black box to decide release quality. It means making test execution policy explicit:
fast targeted checks on feature branches, full validation on integration branches,
and enough observability to explain why a test did or did not run.

CircleCI says testing can consume up to half of CI/CD pipeline time and argues that
a meaningful share of that time is spent running tests unrelated to a change. Its
Smarter Testing suite now packages three controls around that problem: Intelligent
Test Selection, Dynamic Test Splitting, and Auto Rerun Failed Tests. The company
says early users have seen up to four-times faster test runs, with examples including
a backend suite dropping from roughly 12 to 15 minutes per pull request to about
three to five minutes.

## Why slow CI is becoming a platform problem again

Slow CI has always been expensive, but the cost is changing. In older workflows,
a developer might open a pull request after a larger unit of work, wait for a full
suite, and move on. In AI-assisted workflows, developers can generate code, tests,
migrations, and refactors in smaller bursts. That creates more commits, more branch
pushes, and more validation events. A pipeline that was merely annoying under a 
slower development rhythm can become the main throttle on engineering throughput.

The pressure shows up in several ways. Developers context-switch while waiting for
results. Reviewers wait for checks to go green before investing attention. Teams
batch changes because small commits still carry the same CI cost as large ones. 
Flaky tests become more damaging because they interrupt a higher volume of runs.
Platform teams then face a familiar request: add more compute. More compute can 
help, but it does not answer whether the pipeline is running the right work for 
the change under test.

That is where test selection becomes a platform concern. If every service, package,
or monorepo segment invents its own skip rules, confidence erodes. If nobody can
explain skipped tests, release managers will demand full runs everywhere. If the
selection mechanism is too conservative, teams see little benefit. If it is too 
aggressive, defects escape. The job for platform engineering is to turn targeted
testing into a governed service, not a collection of one-off shortcuts.

## How Smarter Testing changes the CI decision model

The most important part of CircleCI’s release is Intelligent Test Selection. It 
uses Test Impact Analysis built from code coverage data to identify tests affected
by changed files. CircleCI describes the selection model as conservative: the command
begins from the full test suite and deselects tests only when coverage data proves
they are unaffected by the change. Full suites still run on the main branch to maintain
the coverage baseline, while feature branches get faster feedback by skipping tests
that should not be relevant.

This distinction matters. Many teams have tried hand-written path filters in CI,
especially in monorepos. Path filters can be useful for coarse routing, but they
often encode ownership assumptions rather than actual execution behavior. A file
in one directory may affect code imported elsewhere. A shared fixture can influence
tests across several packages. Coverage-driven selection is closer to the risk model
teams actually care about: which tests observe the code touched by this change?

Dynamic Test Splitting tackles a different waste pattern. Parallelism does not automatically
mean balanced work. Static splitting based on historical JUnit timing data can leave
one runner overloaded while others sit idle, especially when test startup overhead,
hooks, or environment variance differ from reported test durations. CircleCI’s dynamic
approach uses a shared queue and adapts batch sizes so slower or later-starting 
nodes do less remaining work, reducing the tail latency created by uneven partitions.

Auto Rerun Failed Tests focuses on flakiness. Instead of forcing a developer to 
identify a likely flaky failure and manually rerun a job, the system retries failed
test atoms within configured limits. If a test eventually passes within those limits,
the job can continue without human intervention. If a real defect is present, the
test keeps failing and the job remains failed. The control is not a cure for bad
tests, but it removes avoidable interruption from known intermittent behavior.

## The operating model: fast branches, strict gates

A sensible adoption pattern starts by separating branch feedback from release confidence.
Feature branches should optimize for quick, relevant signal. Main, release, or protected
integration branches should preserve full validation. That gives developers short
loops while keeping a complete recurring baseline for coverage, historical timing,
and audit confidence.

For platform teams, the practical policy might look like this: run selected tests
on pull request updates, run full suites after merge to main, require full suites
before release tags, and run scheduled full validation for packages with heavy transitive
use. Teams with regulated or high-risk services may add stricter gates, but the 
principle is the same. Targeted testing is most defensible when it is paired with
a clear place where the complete suite still runs.

That model also gives test selection data somewhere to refresh. Coverage maps and
timing data are not permanent truths. They age as code changes, tests are renamed,
dependencies shift, and fixtures move. Full main-branch runs are not just safety
checks; they are the source of truth that keeps future selected runs useful. Platform
teams should monitor baseline freshness the same way they monitor cache hit rates
or build image age.

## What to require before turning it on

The first requirement is reliable test metadata. CircleCI’s Intelligent Test Selection
depends on coverage output from the test runner. Most modern ecosystems can produce
it, but platform teams should verify the quality of coverage reports before relying
on them. Missing files, generated code, integration tests that exercise external
systems, and tests that share global setup can all distort the map between code 
and tests.

The second requirement is transparent reporting. Developers need to know why a test
ran, why it was skipped, and when the next full suite will cover the skipped area.
A dashboard that shows skip rates, runtime trends, and per-test history is not just
nice to have. It is how teams build trust in selection. If a production bug appears,
responders need to reconstruct whether the relevant test was selected, skipped, 
flaky, or absent.

The third requirement is an exception path. Some changes should force broader validation
regardless of coverage data. Build scripts, dependency manifests, shared test utilities,
CI configuration, database migration frameworks, authentication libraries, and generated
clients can affect behavior in ways that are hard to model with ordinary unit coverage.
Platform teams should define labels, file patterns, or workflow inputs that force
full suites when the change is structurally risky.

The fourth requirement is cost visibility. Selected testing can reduce compute spend,
but dynamic splitting and paid test analytics may change the billing model. Treat
CI optimization as a cost and productivity project together. Track branch runtime,
main runtime, rerun rates, skipped test percentages, developer wait time, and compute
spend before and after the rollout. Without baseline numbers, teams will argue from
anecdotes.

## Where GitHub Actions fits into the same trend

GitHub’s early September Actions updates show a parallel movement toward more observable
and controllable automation. A new REST API exposes runner version deprecation timelines,
including when runtime support and registration support end for a given runner version.
New reusable workflow context fields identify the workflow that defines a job at
runtime. A new read-only `vulnerability-alerts` permission gives workflows narrower
access to Dependabot alerts.

Those changes are not about test selection, but they reinforce the same platform
pattern. CI/CD systems are becoming policy surfaces. Teams need to know which runner
version is nearing deprecation, which reusable workflow actually defined a job, 
and which workflow permissions are truly required. The more organizations centralize
developer experience on shared automation, the more they need machine-readable control
points instead of tribal knowledge.

For Smarter Testing specifically, this broader trend matters because targeted CI
only works when the surrounding automation is trustworthy. Runner drift, ambiguous
reusable workflow identity, and overbroad tokens can undermine confidence in the
test result. Platform teams should treat CI acceleration and CI governance as one
program. Faster feedback is valuable only if the system producing that feedback 
is understandable and maintainable.

## A rollout plan for platform teams

Start with one high-volume repository where test time is a real developer complaint
and the suite already produces usable coverage. Avoid the most critical release 
path for the pilot. The ideal first candidate is a service or package with frequent
pull requests, localized changes, stable tests, and a main-branch suite that can
remain comprehensive.

Measure the current state for at least a short baseline window. Capture median and
p95 pull request test time, full-suite time on main, queue time, rerun frequency,
flaky test rate, parallel node imbalance, and compute consumption. Then enable the
new test command and selection controls for pull request branches while keeping 
main full. The first target should be observability, not maximum skipping. Teams
should inspect which tests are skipped and compare that against developer intuition.

Use dynamic splitting after selection is stable. If selected test sets are small,
splitting may be less important. If suites remain large or parallel nodes finish
at very different times, dynamic splitting can reduce tail latency. Watch node utilization
and total wall-clock time rather than simply counting how many executors were launched.

Introduce auto reruns with tight limits. A small retry budget can remove noisy interruptions,
but generous retries can hide a test suite that needs repair. Track every auto rerun,
label the most frequent offenders, and keep a backlog for test stabilization. The
best outcome is not a permanently high auto-rerun rate; it is fewer manual interruptions
while the team pays down the worst flaky tests.

## Risks and failure modes

The biggest risk is treating targeted testing as a replacement for test strategy.
If the test suite has poor coverage, weak assertions, or missing integration checks,
selection will make bad signal faster. Platform teams should pair rollout with coverage
review and post-incident learning. When a defect escapes, ask whether the right 
test existed, whether it was selected, whether the coverage map was stale, and whether
a full-suite gate would have caught it.

Another risk is developer confusion. If a pull request shows fewer tests than expected,
engineers may assume the system is broken or gaming the metrics. Clear check names,
analytics links, and repository-level documentation help. The point is not to hide
skipped tests; it is to make them explainable.

There is also an organizational risk. CI optimization projects can become platform-
owned mandates that application teams quietly bypass. A better approach is to publish
a default policy, provide escape hatches for full validation, and review data with
service owners after the pilot. If teams can see shorter waits without losing confidence,
adoption becomes much easier.

## The takeaway

CircleCI’s Smarter Testing release is not just a vendor feature announcement. It
is a reminder that CI/CD systems need to become more selective as development accelerates.
AI-assisted coding raises the volume of changes. Large test suites raise the cost
of validating those changes. Platform teams sit between those forces.

The practical answer is not to run everything all the time or to skip tests blindly.
It is to build a tiered feedback model: selected tests for fast branch feedback,
full suites for integration confidence, dynamic work distribution to reduce idle
compute, bounded retries for known flakes, and enough reporting to explain each 
decision. Done well, that turns CI from a growing tax into a governed developer 
experience service.

## Sources

 * [CircleCI: Smarter Testing: Stop running tests that don’t matter](https://circleci.com/blog/smarter-testing-stop-running-tests-that-dont-matter/)
 * [GitHub Changelog: GitHub Actions early September 2026 updates](https://github.blog/changelog/2026-09-03-github-actions-early-september-2026-updates/)