OpenAI Cuts Habitat Read Latency 12x to 10 ms for 1 Billion ChatGPT Users
Updated
Updated · OpenAI · Oct 23
OpenAI Cuts Habitat Read Latency 12x to 10 ms for 1 Billion ChatGPT Users
1 articles · Updated · OpenAI · Oct 23
Summary
Global read p50 latency on OpenAI’s Habitat storage platform fell from about 120 ms to 10 ms in 2026, improving response times for more than 1 billion ChatGPT users across 40 regions.
The gain came from a layered read strategy: replicating small latency-critical datasets broadly, adding a distributed Valkey cache near users with Cosmos DB change-feed invalidation, and rewriting Habitat service code in Rust.
OpenAI said Habitat now serves more than 500 petabytes of data and has shifted back to multi-tenant Azure Cosmos DB containers by default after Azure added hot-key throttling, QoS and an improved serverless mode.
That redesign reduced provisioning and noisy-neighbor incidents while supporting accounts storing tens of petabytes and handling more than 20 million QPS, with OpenAI and Azure jointly operating the system after major 2024 failover outages.
Looking ahead, OpenAI said it is building an in-house exabyte-scale storage system using RocksDB Cloud and Raft to support the next billion users and reduce flash-capacity needs.
Why is OpenAI abandoning the very Azure database they spent years co-engineering to build a custom exabyte-scale system?
How did a simple Python client secretly evolve into a 500-petabyte monster supporting one billion ChatGPT users?
What hidden breaking points in cloud infrastructure force AI giants to co-manage services instead of just renting them?
Achieving 10ms Latency at 1 Billion User Scale: Inside OpenAI’s Rust-Powered Habitat Overhaul
Overview
OpenAI’s Habitat backend, originally built in Python, faced major performance and scaling issues due to the Global Interpreter Lock, event-loop stalls, and inefficient connection pooling. Rather than rewriting immediately, OpenAI waited until its AI coding models matured. In Q2 2026, just two engineers used Codex and GPT-5.5 to migrate Habitat to Rust, making it dramatically more efficient and reducing latency. This allowed OpenAI to handle more requests on the same hardware and cut operational costs. By also restricting Habitat’s database interface to a simple NoSQL model and offloading analytics via CDC pipelines, OpenAI achieved predictable, high-throughput performance at global scale.