Chalk AI, a platform that builds real-time feature infrastructure for machine learning, today announced Chalk Notebooks—a hosted notebook environment designed for agentic ML workflows. The release frames a question that is becoming central to production AI infrastructure: when an agent can write code, query production data, and retrain models, where does the boundary between agent investigation and human decision-making actually sit?
Production Notebooks, Not Local Sandboxes
Most ML notebooks run on a developer’s laptop or in a managed cloud environment disconnected from production data. Chalk Notebooks invert that assumption. The notebook kernel runs inside the customer’s own cloud, adjacent to their production Chalk deployment. Queries execute against live data sources—application databases, data warehouses, streaming pipelines, and APIs—through a single federated SQL interface.
This matters because the gap between training data and production data is one of the most persistent failure modes in ML systems. A model trained on stale or unrepresentative data will degrade silently until something breaks visibly. Chalk’s approach is to eliminate the abstraction layer: the notebook queries the same deployment that serves production features, using the same resolvers and the same temporal logic.
Point-in-Time Correctness as Infrastructure
One of the more technically significant details in the announcement is point-in-time correctness implemented as a query parameter. In production ML, data leakage—using information that wasn’t available at decision time—is a common and costly mistake. Chalk Notebooks allow queries to specify a timestamp, and the platform guarantees that every feature value returned reflects only what was knowable at that moment.
The demo in the announcement illustrates this concretely: an agent investigating a loan default spike builds a training dataset of 72,837 loans, each computed as of the loan’s origination date. The leak rules are written into the query predicates, not enforced after the fact. The same resolvers that serve real-time predictions carry those predicates, which means training and serving agree by definition.
Branching Production for Model Validation
Chalk Notebooks also support querying both a production deployment and a Chalk branch—a copy of the deployment that serves no production traffic—from the same cell. This allows agents (or engineers) to run A/B comparisons between current production behavior and a candidate model or feature set without modifying the live system.
In the loan default example, the agent deploys the new features, resolvers, and model artifact to a branch, then scores the last four weeks of applications against both production and the branch. Production approved 189 of 190 loans from the problematic merchant segment; the branch approved none. The notebook records both the code and the results, creating an auditable trail from investigation to recommendation.
The Agent Governance Problem
The announcement is explicit about what it calls the “professional negligence” of trusting agents on their word. Agents can miss steps, make judgment errors, and return overconfident answers. When the cost of being wrong is high—and in financial ML it routinely is—validation requires following the reasoning and evidence, not just accepting the conclusion.
Chalk’s governance model reflects this:
- Service tokens are scoped to specific environments
- Model access runs through a budgeted, revocable gateway
- Agents operate in sandboxed compute with controlled egress
- Row and column access policies apply to agent-generated SQL
- Write access is environment-specific: on in development, off in production
The posture is that an agent with production data is a different security concern than an engineer with the same access. The notebook records what the agent did, when, and with what data—creating the reproducibility trail that outlasts the chat session.
Implications for Platform Engineering
For platform teams, Chalk Notebooks represent a specific kind of infrastructure bet: that ML reproducibility is not a data science problem but a systems problem. The notebook is not just a coding environment; it is an audit log, a validation framework, and a boundary between automated investigation and human approval.
The broader question is whether this model generalizes. Feature stores have historically focused on serving speed and consistency. Chalk is adding notebook-driven investigation and agent orchestration to that stack, which raises the complexity ceiling but also the ceiling for what teams can safely automate.
The agent in the demo completes its investigation, writes a recommendation, and stops. The human reviews the notebook and decides whether to ship. That separation—agent investigates, human ships—is the actual architecture pattern here, and it is likely to become a template for other agentic infrastructure platforms.
