OpenAI scales Habitat to 70 million requests per second with 12x faster reads, global caching and Azure Cosmos DB

OpenAI published the second part of its Habitat engineering series on October 23, 2026, detailing an application storage platform supporting products used by more than one billion people weekly, processing over 70 million requests per second and managing more than 500 petabytes of data.

Global median read latency declined from approximately 120 milliseconds to 10 milliseconds, a twelvefold improvement. OpenAI attributes the progress to selective geographic replication, Valkey caching and migration of its service layer from Python to Rust, without isolating each change's individual contribution.

The platform combines Azure Cosmos DB, a narrow application storage API, multi-tenant workload controls, distributed caches, change data capture, Kafka and specialized downstream storage systems. The 12x figure concerns p50 storage read latency, not the end-to-end response time of ChatGPT.

··········

HABITAT'S ROLE IN OPENAI'S APPLICATION INFRASTRUCTURE.

Habitat provides a common online storage interface for ChatGPT, the API, Codex and internal applications, centralizing authorization, data routing, encryption, serialization, caching and connection management.

Originating as a Python client library for GPTs in 2023, Habitat evolved into a standalone service as OpenAI's traffic and geographic footprint expanded. Its workload is predominantly read-heavy: more than 90% of requests are reads.

........

Metric	Reported value	Interpretation
Product audience	Over 1 billion weekly	Across OpenAI products
Traffic	Over 70M requests/s	Storage operations
Managed data	Over 500 PB	Platform footprint
Regions	Approximately 40	Serving locations
Reads	Over 90%	Read-heavy workload

........

A single user interaction can generate multiple Habitat storage operations, so the request rate should not be interpreted as the number of ChatGPT prompts per second.

··········

THE ARCHITECTURE: APPLICATION API, COSMOS DB AND SPECIALIZED SERVICES.

Habitat sits between applications and persistent storage, exposing a deliberately narrow NoSQL API based on objects and edges. It supports direct relationships but not unrestricted graph traversal, which can become expensive across accounts and regions.

Azure Cosmos DB serves major online storage workloads. Valkey provides regional caches; CDC distributes changes; Kafka carries events to downstream consumers; Rockset can provide separate query views for complex workloads.

........

Component	Role	Benefit
Habitat API	Application storage interface	Centralized management
Cosmos DB	Online database	Partitioned persistence
Valkey	Regional cache	Low-latency reads
CDC / Kafka	Change distribution	Invalidation and consumers
Rockset	Secondary query views	Workload isolation

........

Separating online storage from complex analytical queries protects latency-sensitive application requests from expensive operations.

··········

WHY OPENAI FIRST ADOPTED DEDICATED DATABASE CONTAINERS.

Early shared Cosmos DB containers pooled capacity but suffered noisy-neighbor effects. Logical partition keys such as user_id and workspace_id could become hot, exhausting available throughput and throttling unrelated applications.

Cosmos DB measures operations using request units per second (RU/s), and logical partitions share resources through physical partitions. A hot logical key can consume capacity needed by other tenants.

Dedicated containers improved isolation but required individual product teams to predict and provision capacity before launches. Traffic could jump dramatically from testing to production, triggering emergency scaling.

........

Scenario	Shared container	Dedicated container
Provisioned capacity	100M RU/s	10,000 RU/s
Available headroom	20M RU/s	10,000 RU/s
Test demand	1,000 RU/s	1,000 RU/s
Launch demand	1M RU/s	1M RU/s
Demand / headroom	5%	10,000%

........

··········

RETURNING TO MULTI-TENANCY WITH HOT-KEY THROTTLING AND QOS.

OpenAI returned to shared containers by introducing hot-key throttling, request priorities and improved partition scaling. Excessively active keys can be throttled without indiscriminately penalizing unrelated workloads.

Latency-sensitive interactive operations receive priority over background imports and exports. In collaboration with Azure, OpenAI adopted an improved serverless mode supporting up to 10,000 RU/s per physical partition, with splits driven by resource use as well as storage growth.

The company reported no provisioning or noisy-neighbor incidents over the subsequent months. This is a time-bounded operational observation, not a permanent reliability guarantee.

··········

CO-ENGINEERING AZURE COSMOS DB WITH MICROSOFT.

At Habitat's scale, database gateway behavior, SDK connection management and metadata services affected reliability. During regional failovers, rapid client reconnections could overwhelm gateway resources before physical partitions fully recovered.

OpenAI and Microsoft's Azure Cosmos DB engineers developed shared monitoring, production-sized testing, improved metadata caching, connection reuse, account sizing and operational escalation.

The partnership illustrates that hyperscale managed databases can require direct engineering collaboration between customer application teams and the database provider.

··········

GLOBAL MEDIAN READ LATENCY FALLS FROM 120 TO 10 MILLISECONDS.

OpenAI reports a global p50 read latency reduction from approximately 120 ms to 10 ms, representing 12x faster median reads and a 91.7% reduction in latency.

........

Metric	Before	After
Global p50 read latency	~120 ms	~10 ms
Relative speed	1x	12x
Latency reduction	Baseline	~91.7%

........

The gain combines lower network distance for critical datasets, caching of frequently accessed objects and lower processing overhead in the Rust service. OpenAI does not publish a controlled contribution breakdown or a full independent benchmark workload.

The p50 statistic does not establish equivalent improvements for tail latency, writes, uncached reads or complete ChatGPT response time.

··········

SELECTIVE GEOGRAPHIC REPLICATION AND DATA PLACEMENT.

Habitat serves approximately 40 regions, but replicating more than 500 PB everywhere would create substantial storage, network and physical capacity costs.

OpenAI selectively replicates small, latency-sensitive datasets such as authentication metadata. Some accounts are replicated to every serving region, while others maintain at least one replica per continent.

Residency requirements constrain data placement, and an internal routing layer helps move datasets between accounts with different geographic topologies. Larger datasets generally rely more heavily on regional caching.

··········

VALKEY LOOKASIDE CACHES AND CDC INVALIDATION.

With reads exceeding 90% of requests, a distributed lookaside cache avoids repeated cross-region database trips. Valkey servers run close to application compute.

A cache hit serves the object locally; a miss fetches persistent data and populates the cache. Freshness is challenging because short TTLs increase misses while long TTLs risk stale data.

OpenAI uses the Cosmos DB change feed to distribute cache invalidations. When an object changes, CDC consumers notify regional caches, and subsequent reads fetch the updated value. OpenAI reports global invalidation propagation within seconds.

........

Cache strategy	Advantage	Trade-off
Short TTL	Lower staleness	More misses
Long TTL	Higher hit rate	Stale reads
Write invalidation	Direct reaction	Failure races
CDC invalidation	Central propagation	Delivery delay
CDC plus TTL	Bounded staleness	Policy complexity

........

The architecture balances latency and freshness; CDC propagation and replication can still create temporary consistency windows.

··········

THE PYTHON-TO-RUST REWRITE.

Habitat's Python implementation performed serialization, routing, encryption, compression, checksums and connection management on every request. CPU-heavy work could delay asynchronous event-loop scheduling and increase tail latency.

In the second quarter of 2026, two engineers used Codex and GPT-5.5 to rewrite the service in Rust. By the September report, Rust handled about 95% of production traffic.

OpenAI reported sixfold CPU-efficiency and fifteenfold memory-efficiency improvements relative to its Python service. These are workload-specific engineering results, not universal comparisons between programming languages.

No complete financial accounting translates the efficiency changes into monthly infrastructure savings.

··········

CHANGE DATA CAPTURE AT MILLIONS OF EVENTS PER SECOND.

The CDC platform processes millions of changes per second across thousands of Cosmos DB containers and millions of partitions, distributing events through more than 100 Kafka topics.

An internal control plane assigns change-feed readers to regions and workers, handles leases, retries and checkpoints, and isolates failures. Dead-letter queues prevent permanently failing events from blocking unrelated streams.

OpenAI optimized SDK checkpoint behavior by batching writes and can switch readers to alternate replica regions during disruption.

........

Function	Mechanism	Purpose
Change capture	Cosmos DB feed	Detect updates
Assignment	Control plane	Distribute work
Distribution	Kafka	Downstream consumers
Failure handling	Dead-letter queues	Isolate failures
Checkpointing	Batched writes	Reduce overhead

........

Not every downstream consumer has the same latency or consistency guarantees; the public report does not provide a single end-to-end delivery figure for all CDC pipelines.

··········

COSTS, CAPACITY AND MULTI-TENANCY TRADE-OFFS.

Shared containers pool capacity and reduce product-team forecasting burdens but require strong fairness controls. Dedicated containers reduce failure blast radius but can waste provisioned headroom.

Geographic replication lowers latency but multiplies storage and replication requirements. Regional caches add infrastructure and invalidation complexity while avoiding expensive repeated distant reads.

Rust reduces service-layer resource consumption, and CDC centralizes change propagation at the cost of worker and messaging infrastructure.

OpenAI does not publish per-request storage costs, total annual Habitat spending or a quantified overall return on investment. Efficiency gains cannot automatically be interpreted as lower total expenditure.

··········

FUTURE EXABYTE-SCALE STORAGE WITH ROCKSDB CLOUD AND RAFT.

OpenAI is developing an additional online storage backend based on RocksDB Cloud technology and Raft replication, intended for bulky data and eventual exabyte-scale operation.

The design uses disaggregated storage and physical replication while preserving a document model compatible with Cosmos DB. OpenAI expects improved flash-storage economics, with a trade-off in peak query throughput.

The project remains under development, without a published general-availability date, price list or reproducible production benchmarks. It is not a completed replacement for Cosmos DB.

··········

METHODOLOGY, LIMITATIONS AND WHAT THE RESULTS ESTABLISH.

The 12x claim is a global median read-latency comparison, not a universal acceleration factor. The 70M+ requests/s metric represents internal storage operations, and the 500+ PB figure spans the broader platform.

........

Claim	Supported meaning	Limitation
12x reads	Global p50 120→10 ms	Not ChatGPT latency
70M+ requests/s	Storage requests	Not user prompts
500+ PB	Managed storage	Not one container
6x CPU / 15x memory	Rust vs Python service	Implementation-specific
Exabyte storage	Future architecture	Not yet general availability

........

The Rust efficiency gains describe this specific implementation. Months without noisy-neighbor incidents do not establish that such failures are impossible. The future exabyte-scale system is an engineering plan, not an already deployed service.

OpenAI provides useful production measurements and incident-driven engineering details but not sufficient public workload traces to independently reproduce the complete benchmark.

··········

WHAT HABITAT REVEALS ABOUT SCALING AI APPLICATIONS.

Habitat shows how AI application infrastructure must evolve separately from model inference. Authentication, settings and conversation state require storage capable of serving geographically distributed workloads at enormous scale.

OpenAI shifted from shared to dedicated containers and back to shared multi-tenancy as the main constraint moved from isolation to operational complexity. It combined replication, caching and Rust optimization to lower median reads from 120 to 10 ms.

CDC provides a common change-distribution layer, while future bulky datasets may move to a specialized storage engine. The architecture continues to balance reliability, latency, consistency and infrastructure economics rather than maximizing any single metric.

··········

FOLLOW US FOR MORE.

·····

DATA STUDIOS

·····

datastudios.org