# NVIDIA and CrowdStrike Show Why Agentic AI Needs Closed-Loop Evals

- Link: https://thestackobserver.com/nvidia-and-crowdstrike-show-why-agentic-ai-needs-closed-loop-evals/
- Published: 2026-09-11T10:17:31-04:00
- Author: Stackxx

NVIDIA and CrowdStrike have published one of the more concrete demonstrations of
where agentic AI is heading in security: not a chatbot that explains alerts, but
a controlled offense-defense loop where agents generate attacks, reconstruct telemetry,
write detections, validate them, and then force a second attack to adapt. The central
lesson is not that autonomous cyber defense is ready to run unsupervised across 
production networks. It is that the useful unit of progress is shifting from a smarter
model to a measured agent system with tools, grounding, replay, and independent 
review built into the workflow.

The companies evaluated an agentic attack-defense system in an isolated environment
modeled on NVIDIA accelerated computing infrastructure. In the defensive path, Nemotron
models customized for cybersecurity operated within CrowdStrike SafeMind, an agentic
cybersecurity system that brings offensive and defensive AI into the same iterative
loop. The system did not simply ask a model to write detection logic from a prose
description. It gave the agent traces, endpoint telemetry, schema knowledge, validation
tools, and repeated feedback from failed checks.

That distinction matters. Security operations already has plenty of generative AI
features that summarize incidents, draft queries, or explain alerts. The harder 
problem is whether an agent can make a sequence of decisions that improves defensive
coverage without inventing fields, overfitting to a single host, or producing rules
that look reasonable but do not fire on real telemetry. NVIDIA’s write-up is valuable
because it treats those failure modes as first-class engineering constraints.

## What happened

The evaluation connected a red-agent harness and a blue-agent harness inside a representative,
isolated environment. The red side started with a threat-informed objective, selected
an attack path, executed it, and recorded an action trace. CrowdStrike Falcon endpoint
sensors captured the corresponding telemetry. The blue side then received the action
trace, sensor data, and broader attack context, reconstructed what happened, identified
visibility and detection gaps, and generated candidate detections.

The key design choice was the loop. Candidate detections were linted, replayed against
captured telemetry, reviewed, deployed to the detection engine, and then tested 
against independently seeded attacks from the same scenario family. After the defense
improved, the red agent had to adapt with knowledge of the detection context. The
process continued until the modeled environment no longer yielded an obvious viable
path.

In other words, the agent was evaluated less like a single model response and more
like a security engineering workflow. A detection had to pass syntax checks, use
supported fields, avoid brittle environment-specific strings, match recorded telemetry,
stay quiet on available benign traffic, and survive an independent quality review.
These are the kinds of gates a human detection engineering team would normally apply,
but encoded into a repeatable agent harness.

## Why the architecture matters

The defensive harness combined six mechanisms that are quickly becoming table stakes
for high-risk agent deployments. First, a schema knowledge base let agents enumerate
supported Falcon sensor schemas, fields, and query syntax. That reduced a classic
LLM failure mode: writing plausible queries against fields that do not exist. Second,
telemetry grounding anchored the agent in observed traces and sensor events rather
than free-form speculation.

Third, detection authoring was split from broad orchestration. Nemotron 3 Ultra 
handled reconstruction, planning, and tool use, while a customized Nemotron 3 Super
acted as a bounded expert for writing and repairing detections. That separation 
is important because long-horizon orchestration and specialized artifact generation
are different jobs. A single general model can attempt both, but a system that assigns
narrow responsibility to a post-trained expert can be easier to test and improve.

Fourth, artifact linting rejected syntax errors, unsupported fields, and detections
tied too tightly to specific hosts, users, IP addresses, or subnets. Fifth, replay
tested whether a candidate detection actually matched the captured attack. Sixth,
an independent judge reviewed behavioral alignment and robustness. Failed checks
returned structured feedback to the agent, creating a repair loop instead of a one-
shot generation step.

This pattern is bigger than cybersecurity. The more consequential an agent’s work
becomes, the less acceptable it is to trust a fluent final answer. Teams need typed
tools, authoritative context, deterministic checks, realistic replay, and independent
review. In security, the artifact is a detection. In software engineering, it may
be a patch plus tests. In finance, it may be a compliance-sensitive analysis. In
operations, it may be a proposed mitigation. The artifact changes, but the harness
pattern is similar.

## The numbers are promising, but narrow

NVIDIA reports that adding the tuned harness, customized Nemotron 3 Super, domain
context, tools, and validation raised mean backtest detection from 16.5% with Nemotron
3 Ultra and a default harness to 41.9% across evaluated sessions. In live-fire testing,
11 backtest-passing detections from the optimized open pipeline and 35 from a complete
frontier system were deployed against eight unseen attacks from the same scenario
family. Five of the 11 open detections fired on at least one attack, compared with
10 of the 35 frontier detections.

The most interesting result was not raw volume. The frontier system produced more
backtest-passing detections, but the optimized open-model pipeline produced a higher
share that generalized and was the only system in the reported evaluation to produce
detections that met the stricter gold standard. After noise checks and independent
review, three open detections qualified as gold, while none of the frontier detections
did.

Those figures should be read carefully. The evaluation covered one scenario family,
and the detection sets were small. Limited benign traffic means the noise test should
not be interpreted as production false-positive performance. The results also compare
complete systems, not isolated models. The improvement comes from the model stack,
post-training, domain context, tools, validation, and harness design working together.

That is precisely why the work is useful. It pushes the discussion away from leaderboard-
style model comparison and toward system evaluation. For agentic AI, the question
is rarely, “Which model is smartest in isolation?” The practical question is, “Which
combination of model, tools, context, controls, and evals produces reliable work
under realistic constraints?”

## Process supervision is moving from research idea to operating requirement

A second recent thread reinforces the same point. Braintrust and Basis introduced
behavior specs as an open standard for supervising long-horizon agents across a 
trajectory, not only judging the final answer. Their argument is that outcome evals
remain necessary, but they miss whether an agent reached the result through an acceptable
process. In domains such as tax, engineering, security, or operations, a correct-
looking final artifact can still be unacceptable if the agent skipped required evidence,
used the wrong source, or made an unsafe tool call along the way.

That idea maps directly onto the NVIDIA and CrowdStrike evaluation. The system did
not just grade whether a detection eventually appeared. It evaluated whether the
agent used valid schemas, grounded the rule in observed behavior, avoided environment-
specific strings, replayed the detection, and survived review. Those are process
requirements. They define how the agent should behave before anyone trusts the output.

For engineering teams building agents, this is the most important operational lesson.
As agents become longer-running and more autonomous, final-answer scoring becomes
too coarse. The trace is the product. Tool calls, retrieved context, validation 
failures, repairs, approvals, and budget all become evidence. If those events are
not captured and scored, the team cannot tell whether it has improved the agent 
or merely made it more persuasive.

## What changes for security teams

Security teams should not read this as a cue to hand detection engineering to autonomous
agents wholesale. The near-term opportunity is narrower and more practical: use 
agents to accelerate closed-loop testing in controlled environments where the blast
radius is bounded and the ground truth is observable.

A realistic adoption path starts with representative cyber ranges, historical incidents,
or sanitized infrastructure models. Red-agent activity can generate diverse traces.
Blue-agent workflows can propose detections. Deterministic validators can reject
invalid syntax, unsupported fields, brittle identifiers, and missing telemetry matches.
Human reviewers can focus on the smaller set of candidates that pass mechanical 
checks. Over time, production feedback can become new eval cases.

The biggest organizational shift is that detection engineering becomes more like
continuous integration. A candidate rule is not “done” because a model wrote it 
or because it matched one trace. It is done when it passes a battery of checks, 
detects unseen variants, avoids noise, and remains explainable to the human team
responsible for the environment. Agents can make that loop faster, but they also
make the evaluation discipline more important.

## What changes for agent builders

For agent platform teams, the architecture suggests four practical design principles.

 * **Separate orchestration from specialist work.** A reasoning model can manage
   context and tools, while a smaller or customized model handles bounded artifact
   generation.
 * **Give agents authoritative interfaces.** Schema browsers, typed tools, and documented
   constraints reduce hallucinated operations more reliably than reminders in a 
   prompt.
 * **Validate artifacts deterministically before judging quality.** Syntax checks,
   linting, replay, unit tests, or policy checks should catch obvious failures before
   expensive review.
 * **Evaluate full trajectories.** A correct output reached through the wrong process
   is a latent incident. Capture the trace and score the behaviors that matter.

These principles also explain why agentic AI is becoming an infrastructure problem.
The hard part is no longer only prompting the model. It is building the harness 
around the model: sandboxes, memory, skills, tool registries, approval policies,
replay systems, eval datasets, and observability. The agent needs enough freedom
to solve the task, but enough structure that its work can be inspected, repeated,
and improved.

## The broader signal

The NVIDIA and CrowdStrike work arrives as agent evaluation is becoming a more serious
discipline. Braintrust’s behavior specs focus on process-level supervision. NVIDIA’s
SkillEvaluator work, published separately, measures whether packaged skills improve
agent performance by comparing runs with and without the skill in isolated harnesses.
These are different projects, but they point in the same direction: agent quality
is becoming measurable at the level of workflow, context, and repeatable behavior.

That is the right direction for the market. The next phase of agentic AI will not
be won only by models that can reason longer. It will be won by systems that know
when to retrieve, when to act, when to stop, when to ask for review, and how to 
prove that a proposed artifact works. Cybersecurity is a natural proving ground 
because the stakes are high, the adversary adapts, and the telemetry can be replayed.
If closed-loop agents can make progress there under controlled conditions, the same
design language will spread into other operational domains.

The caveat is equally important. A closed-loop benchmark is not production autonomy.
Real enterprise environments have messier telemetry, more varied attack paths, compliance
constraints, changing schemas, and political risk around false positives and automated
response. The right takeaway is not that security teams can remove humans from the
loop. It is that humans can move up the loop, defining scenarios, reviewing gold
candidates, setting policy, and deciding when validated agent output is ready to
ship.

## What to watch next

The next meaningful milestones will be broader evaluations across multiple scenario
families, richer benign-traffic tests, clearer confidence intervals, and evidence
that agent-generated detections can survive production drift. It will also matter
whether teams can package these patterns into reusable harnesses instead of bespoke
demos. If every organization must rebuild the loop from scratch, adoption will be
slow. If schema grounding, replay, behavior specs, and trace-level evals become 
standard components, specialized agents will become much easier to trust.

For now, the strongest signal is conceptual. Agentic AI is moving from “model plus
tools” to “model plus measured operating system.” NVIDIA and CrowdStrike’s cybersecurity
loop shows what that looks like in a demanding domain: agents are useful when their
decisions are grounded, their artifacts are tested, and their failures feed the 
next iteration.

## Sources

 * [NVIDIA Technical Blog: Building an Adaptive Agentic Cybersecurity System with NVIDIA Nemotron](https://developer.nvidia.com/blog/building-an-adaptive-agentic-cybersecurity-system-with-nvidia-nemotron/)
 * [Braintrust: Behavior specs, an open standard for supervising long-horizon agents](https://www.braintrust.dev/blog/behavior-specs)
 * [NVIDIA Technical Blog: Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/)