OpenAI has detailed how it upgraded its online storage infrastructure to support ChatGPT and other products as the company serves more than 1 billion users. In a technical article published on October 23, 2026, OpenAI explained how it improved Habitat, its internal storage platform, through changes to Microsoft Azure Cosmos DB, distributed caching, multi-tenancy and global data placement. The company reported that these improvements reduced median global read latency from approximately 120 milliseconds to 10 milliseconds, representing a 12× improvement.
The work addresses a challenge that becomes increasingly difficult as an AI platform grows: delivering user data quickly and reliably across regions while supporting enormous request volumes. Habitat stores and retrieves information used by OpenAI products, including ChatGPT, the API and Codex. By combining database improvements with caching and service-level optimizations, OpenAI aims to make its infrastructure faster and easier for product teams to develop on without requiring every team to build its own storage optimizations.
Key Takeaways
- 12× faster reads: OpenAI reduced global median read latency from roughly 120 milliseconds to roughly 10 milliseconds.
- More than 1 billion users: Habitat is designed to support OpenAI products at this scale across approximately 40 geographic regions.
- Azure Cosmos DB: OpenAI worked closely with Microsoft Azure engineers to improve multi-tenancy, scaling and database reliability.
- Distributed caching: A cache invalidation system keeps frequently accessed data closer to application servers while helping prevent outdated information from persisting.
- Rust optimization: OpenAI has also rewritten parts of Habitat in Rust to reduce service overhead.
- Massive storage footprint: The platform supports more than 500 petabytes of data, according to OpenAI’s earlier technical disclosure.
- Future plans: OpenAI is developing a custom exabyte-scale storage system for its next stage of growth.
What Is Habitat, OpenAI’s Internal Storage Platform?
Habitat is the storage infrastructure OpenAI uses to provide fast, reliable access to data across its products and internal services. Rather than requiring each application team to manage database connections, caching, authorization and data placement independently, Habitat provides a shared platform for these responsibilities.
This architecture is particularly important for AI applications. ChatGPT and other OpenAI services need to retrieve information such as user settings, application state and other stored objects while processing requests. Even a small delay in storage can become noticeable when it occurs repeatedly across a large number of interactions.
OpenAI has designed Habitat around a relatively narrow data model based on objects and their relationships, rather than supporting unrestricted database queries. The company says this constrained approach helps it partition data and scale the service more predictably.
In its September 2026 technical article, OpenAI reported that Habitat was handling more than 70 million requests per second and storing over 500 petabytes of data. Those figures describe the platform at the time of that earlier publication, rather than independently verified measurements of its current workload.
How OpenAI Achieved 12× Faster Reads
OpenAI said it reduced median global read latency from approximately 120 milliseconds to 10 milliseconds by combining three major changes: placing critical data closer to users, introducing distributed caching and reducing overhead in the Habitat service.
| Optimization | What changed | Expected benefit |
|---|---|---|
| Regional data placement | Replicated selected latency-sensitive datasets closer to application traffic | Reduced cross-region communication |
| Distributed caching | Stored frequently accessed data in nearby Valkey cache servers | Fewer requests reaching the underlying database |
| Cache invalidation | Used Azure Cosmos DB change-feed events to invalidate cached entries | Improved freshness without relying only on short expiration times |
| Rust rewrite | Reimplemented parts of Habitat in a more performance-oriented language | Lower service overhead and more efficient resource use |
The 12× figure represents the improvement in median, or p50, read latency. It does not mean every database request is 12 times faster, nor does it establish that all OpenAI products have experienced the same reduction in overall response time.
1. Moving Critical Data Closer to Users
When an application server and its database are in different geographic regions, a request must travel across the network to retrieve information. At global scale, these additional network round trips can significantly increase latency.
OpenAI said it replicates selected datasets to nearby regions when capacity and data-residency requirements permit. Some highly latency-sensitive information, such as authentication metadata, can be replicated across many regions to make it available closer to incoming requests.
However, copying every dataset everywhere is expensive and constrained by physical data-center capacity. OpenAI therefore prioritizes data whose performance benefits justify the additional storage and replication requirements.
2. Using Distributed Caching to Reduce Database Reads
Caching allows an application to keep copies of frequently requested information in faster, nearby infrastructure. If a request can be served from a cache, the system may avoid a slower trip to the underlying database.
OpenAI integrated distributed caching through Valkey servers located close to where application requests are processed. This approach is particularly useful because Habitat’s workload is heavily read-oriented: applications repeatedly retrieve settings, state and other stored properties.
The challenge is keeping cached data accurate. A long time-to-live can improve cache hit rates but may leave outdated information available for too long. A short time-to-live reduces the period of potential staleness but causes more requests to return to the database.
OpenAI’s solution uses change data capture through Azure Cosmos DB’s change feed. When an underlying object changes, a service processes the update and invalidates relevant cache entries globally within seconds. The next read can then fetch the updated version and repopulate the cache.
This helps balance performance with data freshness, although distributed systems still need to handle failures and delays in propagating updates.
3. Rewriting Service Components in Rust
OpenAI has also reduced overhead inside Habitat by rewriting parts of the service in Rust, a programming language known for memory safety and efficient execution.
In its earlier September 2026 article, OpenAI said that Rust handled 95% of production requests at that point, with reported gains of six times in CPU efficiency and 15 times in memory efficiency compared with the relevant earlier implementation.
Those figures relate to the service rewrite and should not be confused with the 12× read-latency improvement. The latest performance result comes from combining several changes across data placement, caching and service execution.
Why Azure Cosmos DB Was Central to the Upgrade
Azure Cosmos DB is Microsoft’s globally distributed database service. OpenAI uses it as the underlying storage component for much of Habitat.
At OpenAI’s scale, however, simply using a managed database is not enough. Workloads can become unusually large, hot data partitions can experience contention, and regional failures can produce connection surges during failover.
OpenAI said it worked with Azure engineers to improve multi-tenancy, request prioritization, partition scaling and gateway behavior. These changes were intended to prevent one busy workload from consuming resources needed by higher-priority interactive requests.
The company also described a shared operational model with Azure that includes joint testing, monitoring and investigation of failure modes. This collaboration became important after earlier regional outages exposed weaknesses in failover procedures and connection handling at very large scale.
OpenAI said the newer shared-container approach simplified capacity management while preserving workload isolation. It also reported that improved serverless scaling helped reduce the need for manual capacity adjustments.
The case illustrates an important point about cloud infrastructure: at extreme scale, performance depends not only on database technology but also on how the customer and provider operate, test and tune the system together.
What This Means for ChatGPT Users and Developers
For ChatGPT users, lower storage latency can help the service retrieve necessary data more quickly. However, a 12× improvement in database read latency does not translate directly into ChatGPT responses being 12 times faster. Model inference, network communication, tool execution and other processing stages also contribute to the time a user experiences.
For developers using OpenAI’s API and internal product teams building on Habitat, the broader benefit is infrastructure that handles more complexity centrally. If the platform manages caching, data placement and authorization effectively, application teams can spend less time implementing workarounds and more time building features.
OpenAI said that earlier limitations had encouraged product developers to add their own caching, batch database requests and move work away from critical paths. While these approaches could improve resilience, they also made applications more complicated. Better storage performance helps reduce the need for such application-specific measures.
OpenAI’s Next Step: Exabyte-Scale Storage
OpenAI said it is developing a custom online storage system to support the next stage of growth. The system is designed for exabyte-scale data volumes and is intended to complement or eventually support workloads that are difficult to serve efficiently with existing storage arrangements.
According to OpenAI, the design combines Rockset’s RocksDB Cloud core with Raft-based replication. It uses a document model similar to Azure Cosmos DB but adopts disaggregated storage and physical replication to reduce the amount of flash capacity required.
This approach involves trade-offs. OpenAI says the design reduces flash-storage requirements in exchange for a lower peak queries-per-second capacity, making it potentially suitable for large, bulky datasets that do not require the same peak request rate as the most latency-sensitive workloads.
The system is under development. OpenAI has not announced a general availability date or claimed that it has already replaced Habitat’s existing storage infrastructure.
The Bigger Picture
OpenAI’s storage improvements demonstrate how infrastructure engineering becomes a competitive factor as AI services expand. Model quality is only one part of delivering an effective AI product. The platform must also retrieve information reliably, manage user state, handle global traffic and recover from failures without imposing unnecessary delays.
The 12× read-latency improvement shows the potential impact of optimizing several layers together rather than relying on a single database upgrade. Regional replication, cache invalidation, workload isolation and efficient service code all contribute to the result.
The longer-term challenge will be balancing performance, cost, reliability and data-residency requirements as OpenAI’s products continue to grow. Its planned exabyte-scale storage system suggests that the company is preparing for further increases in both the volume of stored data and the complexity of its AI workloads.
FAQs
1. What is OpenAI’s Habitat storage platform?
Habitat is OpenAI’s internal storage platform that provides data access, caching, authorization and related infrastructure for products such as ChatGPT, Codex and the OpenAI API.
2. How did OpenAI make storage reads 12 times faster?
OpenAI combined regional placement of critical data, distributed caching with change-feed-based invalidation and performance improvements to the Habitat service, including a rewrite in Rust. The company reported that global median read latency fell from roughly 120 milliseconds to 10 milliseconds.
3. What role does Azure Cosmos DB play?
Azure Cosmos DB provides the underlying database technology used by Habitat. OpenAI worked with Microsoft Azure engineers to improve multi-tenancy, scaling, reliability and performance for its unusually large workloads.
4. Does this mean ChatGPT responses are now 12 times faster?
No. The 12× figure refers specifically to median global storage-read latency. Overall ChatGPT response times also depend on model processing, networking, tool use and other components.
5. What is OpenAI’s exabyte-scale storage project?
OpenAI is developing a custom storage system designed for very large datasets. The company says it combines Rockset’s RocksDB Cloud core with Raft-based replication and disaggregated storage. The system remains under development.
Looking Ahead
OpenAI’s next challenge is to scale storage while keeping data close enough to users for fast access, maintaining reliability during regional disruptions and controlling the cost of storing hundreds of petabytes or more. The company’s partnership with Azure and its continued work on caching and data placement will be central to those goals.
The planned exabyte-scale storage system signals that OpenAI is preparing for further growth beyond its current architecture. If the system delivers the intended storage efficiency while meeting workload requirements, it could give the company more flexibility in supporting future ChatGPT features, API traffic and other data-intensive AI services.
Get the day’s top stories in your inbox
One concise email. No spam, unsubscribe anytime.
