OpenAI’s latest infrastructure write-up is not about a new model, a benchmark win, or a GPU cluster. It is about a storage layer. That is exactly why it matters.
In a September engineering post, OpenAI described Habitat, the online storage platform behind fast access to product data across ChatGPT, Codex settings, login flows, GPTs, and other product surfaces. The headline numbers are large even by hyperscale standards: more than 70 million requests per second, more than 500 petabytes of data, products used by over 1 billion people each week, and traffic spread across almost 40 geographic regions. But the more useful signal is the sequence of choices OpenAI says it made under pressure: start with a Python client library, centralize control into a service when coordination became dangerous, tune the event loop and connection pools instead of pretending Python was free, constrain the API so request cost stayed predictable, then rewrite the hot path in Rust once the platform contract had stabilized.
The lesson for AI infrastructure teams is this: the hard part of scaling AI products is no longer limited to model serving. Once assistants, agents, custom GPTs, memory, files, evaluations, and developer tools become daily workflows, the supporting data plane becomes part of the user-visible inference path. A model can be fast, but an application still feels slow if every request fans out into hundreds of storage calls with high tail latency.
What Happened
OpenAI published the first part of a two-part account of how it scaled Habitat, its online storage platform, to support ChatGPT-scale product traffic. Habitat began around DevDay 2023 as a Python client-side library connected to Azure Cosmos DB. Its purpose was to hide database mechanics from product engineers: schema lookup, routing, authorization, encryption, serialization, request shaping, connection pooling, cache access, and the choice of underlying storage could all sit behind one API.
That library approach helped product teams move quickly. It also became brittle as the company and product surface grew. By mid-2025, OpenAI says Habitat had reached the limits of a client-side implementation. A change intended to reduce the blast radius of regional outages required new routing logic, feature flags, shadowing, bug fixes, and coordinated rollouts across dozens of services. One team rolling back to an older client could reintroduce the very bug the platform team had tried to retire.
The response was to turn Habitat into a standalone service. Centralizing the layer gave OpenAI one place to deploy routing changes, observe system behavior, enforce access control, audit usage, and limit direct access to storage resources. It also turned an ordinary library upgrade problem into a platform control problem: when the service changes, every product can benefit without every product team shipping a new client at exactly the right time.
That move came with a cost. Habitat was still written in Python, a language OpenAI describes as strategically useful but inefficient for a high-throughput serving layer. Python’s asyncio model helped with I/O concurrency, but it did not remove CPU contention or the global interpreter lock. Habitat was doing work that looks small in isolation and large at scale: routing, compression, encryption, checksumming, downstream health checking, request shadowing, and hedging. At high utilization, OpenAI found that event-loop scheduling delay could dominate p99 and worse latencies even when the downstream database had already responded.
Why This Matters Now
The timing is important because the industry conversation about AI infrastructure is still heavily weighted toward accelerators, model routers, and inference engines. Those systems matter, but production AI products are increasingly composite applications. A single user interaction can touch identity, policy, memory, personalization, file metadata, conversation state, tool configuration, evaluation logs, billing limits, and model routing. The model call is only one leg of the trip.
Habitat shows how quickly a platform can move from convenience abstraction to critical infrastructure. At modest scale, a shared library is often the fastest route to consistency. At ChatGPT scale, the same library becomes an operational liability if every change requires dozens of teams to roll forward in lockstep. The problem is not that libraries are bad. The problem is that a library cannot enforce global behavior after it has already been copied into every service’s deployment timeline.
That distinction matters for enterprises building internal AI platforms. Many teams begin with SDKs around vector databases, object storage, feature stores, prompt stores, policy engines, or model gateways. The SDKs make adoption easy. But if those SDKs become the only place where routing, security, privacy, quota, and failover behavior lives, the organization inherits the same rollout problem OpenAI described. Every safety fix, region move, or data placement change becomes a distributed coordination exercise.
The Habitat story also underlines an uncomfortable truth about language and stack choices. OpenAI did not frame Python as a mistake. It framed Python as a deliberate debt instrument: useful for establishing the API and unblocking product teams, expensive at scale, and worth replacing after the contract was proven. That is a more mature lesson than simply saying high-throughput services should be written in Rust or Go from day one. Premature rewrites can freeze an API too early; late rewrites can trap the company in runaway infrastructure cost. The tradeoff is timing.
The Tail Latency Details Are The Real Story
The most practitioner-relevant part of OpenAI’s post is not the final Rust rewrite. It is the list of operational details that made the Python service survivable while the platform matured.
First, OpenAI measured asyncio scheduling delay directly. Standard service metrics such as CPU, memory, disk, and network utilization were not enough. The team scheduled background tasks and measured the gap between expected and actual execution time to see how busy the event loop really was. Under load, expensive CPU work and background tasks could delay coroutine rescheduling by hundreds of milliseconds, and in edge cases by seconds. For a product request that may depend on many storage calls, those delays become user-visible.
Second, the team found a tail-latency source in feature flag configuration parsing. Statsig polling was refreshing large production rules every minute, without enough jitter, across multiple Python processes per pod. The result was synchronized CPU work that stalled in-flight request processing. The fix was not exotic: smaller targeted configs, longer refresh intervals, and jitter. The lesson is sharper than the fix. Control-plane conveniences can become data-plane latency events when they run inside the hot path at scale.
Third, OpenAI traced a metastable load-balancing failure to connection reuse behavior. Python’s aiohttp TCPConnector used LIFO connection reuse, meaning recently returned connections were favored for the next request. After a burst, slower overloaded servers returned connections later, making them more likely to receive more traffic and become even slower. Patching the pool to use FIFO helped break that feedback loop. OpenAI says it now mostly depends on Istio and Envoy for connection pooling and load-aware balancing, but the deeper point is that defaults in a client connector can shape fleet-level behavior.
Fourth, the company used Envoy to reduce downstream connection pressure. Scaling out many Python processes helped keep event-loop delay low, but it also created a risk of overwhelming downstream systems with too many connections. Envoy let OpenAI fan in connections, upgrade Python HTTP/1 traffic to HTTP/2, use multiplexing, extend connection lifetimes, and centralize rate limits and circuit breakers. That is the kind of glue work that rarely appears in AI product demos but often determines whether the product stays up.
Habitat’s API Is A Constraint, Not Just An Abstraction
OpenAI also emphasized what Habitat does not allow. It exposes a constrained NoSQL API designed around predictable, constant-work requests. It does not let clients issue arbitrary SQL queries, large table scans, complex joins, or graph traversals against the online storage layer. That constraint is not merely an implementation detail. It is a scalability strategy.
In fast-growing systems, cost imbalance is dangerous. It is easy for a product engineer to write a query that is cheap to express and expensive to run. At small scale, review can catch many of those mistakes. At large scale, the combination of team growth, product growth, and hot paths makes manual query governance weaker. Habitat’s API makes expensive access patterns obvious and pushes complex analytical or search workloads into an offline secondary view using change data capture and isolated Rockset instances.
This design has a product cost. Teams that need richer queries must do more work. But that friction protects the online path. For AI applications, where user requests may already involve model calls, retrieval, tool use, and policy checks, preserving predictable storage behavior can be more valuable than giving every team maximal query flexibility.
The Rust Rewrite Is The Result, Not The Lesson
OpenAI says Habitat’s Python service peaked at more than 20 million requests per second. In the second quarter of 2026, two engineers using Codex and GPT-5.5 rewrote the service in Rust. The Rust version now handles 95% of production requests, with Python slated for deprecation. OpenAI reports that the Rust service is 6x more CPU efficient and 15x more memory efficient than the Python version, with lower average and tail latency.
Those numbers are striking, but the sequencing is the more important part. The rewrite happened after the team had centralized the storage layer, learned the failure modes, narrowed the API, and understood where Python was burning resources. In other words, Rust made the service cheaper and faster after the platform boundary was already clear.
For teams reading this as a migration template, the lesson is not to rewrite every Python AI service. It is to know which parts of the AI product stack are becoming shared control points, which request paths are sensitive to tail latency, and which abstractions have stabilized enough to justify a lower-level implementation. A rewrite is easier to defend when it removes measured cost from a platform used by every product, not when it expresses a general preference for a systems language.
Who Is Affected
The direct audience is platform teams building internal AI infrastructure. If your organization is standardizing model gateways, vector stores, memory layers, or agent runtimes, Habitat is a reminder to think beyond adoption. The easy path for initial adoption may be an SDK. The durable path for governance, observability, security, and coordinated change may need a service boundary.
Application teams are affected too. A platform that constrains queries, routes requests centrally, and pushes analytics off the online path can feel less flexible at first. But it can also make the product more reliable under growth. The trade is between local freedom and global predictability.
Infrastructure vendors should read Habitat as another sign that AI platforms are converging with older distributed-systems concerns. GPU supply, inference throughput, and model quality remain central. But storage fanout, event-loop delay, connection pooling, service mesh behavior, feature flag load, and API cost predictability are now part of the AI product experience.
What Changes Next
OpenAI says a second post will cover multi-tenancy reliability, read-performance optimization, and its Azure Cosmos DB partnership. That follow-up may be even more relevant for teams deciding how much of their AI data plane should live in managed cloud systems versus custom platform services.
For now, Habitat makes one point clearly: AI infrastructure is becoming less special in the places where reliability matters most. The workloads are new, the growth curve is unusual, and the products depend on frontier models. But the engineering work includes familiar decisions about blast radius, centralization, predictable APIs, latency measurement, connection management, and rewriting hot paths only after the shape of the system is known.
The next generation of AI products will not be won only by teams with the largest models or the most GPUs. It will also be won by teams that can make every supporting system boring enough to survive extraordinary demand.
