NVIDIA and CrowdStrike have published one of the more concrete demonstrations of where agentic AI is heading in security: not a chatbot that explains alerts, but a controlled offense-defense loop where agents generate attacks, reconstruct telemetry, write detections, validate them, and then force a second attack to adapt. The central lesson is not that autonomous cyber defense is ready to run unsupervised across production networks. It is that the useful unit of progress is shifting from a smarter model to a measured agent system with tools, grounding, replay, and independent review built into the workflow.
The companies evaluated an agentic attack-defense system in an isolated environment modeled on NVIDIA accelerated computing infrastructure. In the defensive path, Nemotron models customized for cybersecurity operated within CrowdStrike SafeMind, an agentic cybersecurity system that brings offensive and defensive AI into the same iterative loop. The system did not simply ask a model to write detection logic from a prose description. It gave the agent traces, endpoint telemetry, schema knowledge, validation tools, and repeated feedback from failed checks.
That distinction matters. Security operations already has plenty of generative AI features that summarize incidents, draft queries, or explain alerts. The harder problem is whether an agent can make a sequence of decisions that improves defensive coverage without inventing fields, overfitting to a single host, or producing rules that look reasonable but do not fire on real telemetry. NVIDIA’s write-up is valuable because it treats those failure modes as first-class engineering constraints.
What happened
The evaluation connected a red-agent harness and a blue-agent harness inside a representative, isolated environment. The red side started with a threat-informed objective, selected an attack path, executed it, and recorded an action trace. CrowdStrike Falcon endpoint sensors captured the corresponding telemetry. The blue side then received the action trace, sensor data, and broader attack context, reconstructed what happened, identified visibility and detection gaps, and generated candidate detections.
The key design choice was the loop. Candidate detections were linted, replayed against captured telemetry, reviewed, deployed to the detection engine, and then tested against independently seeded attacks from the same scenario family. After the defense improved, the red agent had to adapt with knowledge of the detection context. The process continued until the modeled environment no longer yielded an obvious viable path.
In other words, the agent was evaluated less like a single model response and more like a security engineering workflow. A detection had to pass syntax checks, use supported fields, avoid brittle environment-specific strings, match recorded telemetry, stay quiet on available benign traffic, and survive an independent quality review. These are the kinds of gates a human detection engineering team would normally apply, but encoded into a repeatable agent harness.
Why the architecture matters
The defensive harness combined six mechanisms that are quickly becoming table stakes for high-risk agent deployments. First, a schema knowledge base let agents enumerate supported Falcon sensor schemas, fields, and query syntax. That reduced a classic LLM failure mode: writing plausible queries against fields that do not exist. Second, telemetry grounding anchored the agent in observed traces and sensor events rather than free-form speculation.
Third, detection authoring was split from broad orchestration. Nemotron 3 Ultra handled reconstruction, planning, and tool use, while a customized Nemotron 3 Super acted as a bounded expert for writing and repairing detections. That separation is important because long-horizon orchestration and specialized artifact generation are different jobs. A single general model can attempt both, but a system that assigns narrow responsibility to a post-trained expert can be easier to test and improve.
Fourth, artifact linting rejected syntax errors, unsupported fields, and detections tied too tightly to specific hosts, users, IP addresses, or subnets. Fifth, replay tested whether a candidate detection actually matched the captured attack. Sixth, an independent judge reviewed behavioral alignment and robustness. Failed checks returned structured feedback to the agent, creating a repair loop instead of a one-shot generation step.
This pattern is bigger than cybersecurity. The more consequential an agent’s work becomes, the less acceptable it is to trust a fluent final answer. Teams need typed tools, authoritative context, deterministic checks, realistic replay, and independent review. In security, the artifact is a detection. In software engineering, it may be a patch plus tests. In finance, it may be a compliance-sensitive analysis. In operations, it may be a proposed mitigation. The artifact changes, but the harness pattern is similar.
The numbers are promising, but narrow
NVIDIA reports that adding the tuned harness, customized Nemotron 3 Super, domain context, tools, and validation raised mean backtest detection from 16.5% with Nemotron 3 Ultra and a default harness to 41.9% across evaluated sessions. In live-fire testing, 11 backtest-passing detections from the optimized open pipeline and 35 from a complete frontier system were deployed against eight unseen attacks from the same scenario family. Five of the 11 open detections fired on at least one attack, compared with 10 of the 35 frontier detections.
The most interesting result was not raw volume. The frontier system produced more backtest-passing detections, but the optimized open-model pipeline produced a higher share that generalized and was the only system in the reported evaluation to produce detections that met the stricter gold standard. After noise checks and independent review, three open detections qualified as gold, while none of the frontier detections did.
Those figures should be read carefully. The evaluation covered one scenario family, and the detection sets were small. Limited benign traffic means the noise test should not be interpreted as production false-positive performance. The results also compare complete systems, not isolated models. The improvement comes from the model stack, post-training, domain context, tools, validation, and harness design working together.
That is precisely why the work is useful. It pushes the discussion away from leaderboard-style model comparison and toward system evaluation. For agentic AI, the question is rarely, “Which model is smartest in isolation?” The practical question is, “Which combination of model, tools, context, controls, and evals produces reliable work under realistic constraints?”
Process supervision is moving from research idea to operating requirement
A second recent thread reinforces the same point. Braintrust and Basis introduced behavior specs as an open standard for supervising long-horizon agents across a trajectory, not only judging the final answer. Their argument is that outcome evals remain necessary, but they miss whether an agent reached the result through an acceptable process. In domains such as tax, engineering, security, or operations, a correct-looking final artifact can still be unacceptable if the agent skipped required evidence, used the wrong source, or made an unsafe tool call along the way.
That idea maps directly onto the NVIDIA and CrowdStrike evaluation. The system did not just grade whether a detection eventually appeared. It evaluated whether the agent used valid schemas, grounded the rule in observed behavior, avoided environment-specific strings, replayed the detection, and survived review. Those are process requirements. They define how the agent should behave before anyone trusts the output.
For engineering teams building agents, this is the most important operational lesson. As agents become longer-running and more autonomous, final-answer scoring becomes too coarse. The trace is the product. Tool calls, retrieved context, validation failures, repairs, approvals, and budget all become evidence. If those events are not captured and scored, the team cannot tell whether it has improved the agent or merely made it more persuasive.
What changes for security teams
Security teams should not read this as a cue to hand detection engineering to autonomous agents wholesale. The near-term opportunity is narrower and more practical: use agents to accelerate closed-loop testing in controlled environments where the blast radius is bounded and the ground truth is observable.
A realistic adoption path starts with representative cyber ranges, historical incidents, or sanitized infrastructure models. Red-agent activity can generate diverse traces. Blue-agent workflows can propose detections. Deterministic validators can reject invalid syntax, unsupported fields, brittle identifiers, and missing telemetry matches. Human reviewers can focus on the smaller set of candidates that pass mechanical checks. Over time, production feedback can become new eval cases.
The biggest organizational shift is that detection engineering becomes more like continuous integration. A candidate rule is not “done” because a model wrote it or because it matched one trace. It is done when it passes a battery of checks, detects unseen variants, avoids noise, and remains explainable to the human team responsible for the environment. Agents can make that loop faster, but they also make the evaluation discipline more important.
What changes for agent builders
For agent platform teams, the architecture suggests four practical design principles.
- Separate orchestration from specialist work. A reasoning model can manage context and tools, while a smaller or customized model handles bounded artifact generation.
- Give agents authoritative interfaces. Schema browsers, typed tools, and documented constraints reduce hallucinated operations more reliably than reminders in a prompt.
- Validate artifacts deterministically before judging quality. Syntax checks, linting, replay, unit tests, or policy checks should catch obvious failures before expensive review.
- Evaluate full trajectories. A correct output reached through the wrong process is a latent incident. Capture the trace and score the behaviors that matter.
These principles also explain why agentic AI is becoming an infrastructure problem. The hard part is no longer only prompting the model. It is building the harness around the model: sandboxes, memory, skills, tool registries, approval policies, replay systems, eval datasets, and observability. The agent needs enough freedom to solve the task, but enough structure that its work can be inspected, repeated, and improved.
The broader signal
The NVIDIA and CrowdStrike work arrives as agent evaluation is becoming a more serious discipline. Braintrust’s behavior specs focus on process-level supervision. NVIDIA’s SkillEvaluator work, published separately, measures whether packaged skills improve agent performance by comparing runs with and without the skill in isolated harnesses. These are different projects, but they point in the same direction: agent quality is becoming measurable at the level of workflow, context, and repeatable behavior.
That is the right direction for the market. The next phase of agentic AI will not be won only by models that can reason longer. It will be won by systems that know when to retrieve, when to act, when to stop, when to ask for review, and how to prove that a proposed artifact works. Cybersecurity is a natural proving ground because the stakes are high, the adversary adapts, and the telemetry can be replayed. If closed-loop agents can make progress there under controlled conditions, the same design language will spread into other operational domains.
The caveat is equally important. A closed-loop benchmark is not production autonomy. Real enterprise environments have messier telemetry, more varied attack paths, compliance constraints, changing schemas, and political risk around false positives and automated response. The right takeaway is not that security teams can remove humans from the loop. It is that humans can move up the loop, defining scenarios, reviewing gold candidates, setting policy, and deciding when validated agent output is ready to ship.
What to watch next
The next meaningful milestones will be broader evaluations across multiple scenario families, richer benign-traffic tests, clearer confidence intervals, and evidence that agent-generated detections can survive production drift. It will also matter whether teams can package these patterns into reusable harnesses instead of bespoke demos. If every organization must rebuild the loop from scratch, adoption will be slow. If schema grounding, replay, behavior specs, and trace-level evals become standard components, specialized agents will become much easier to trust.
For now, the strongest signal is conceptual. Agentic AI is moving from “model plus tools” to “model plus measured operating system.” NVIDIA and CrowdStrike’s cybersecurity loop shows what that looks like in a demanding domain: agents are useful when their decisions are grounded, their artifacts are tested, and their failures feed the next iteration.
