Google Introduces LLM Sleep Framework, Retaining Knowledge Across 10M Tokens
Updated
Updated · Google Research · Oct 27
Google Introduces LLM Sleep Framework, Retaining Knowledge Across 10M Tokens
1 articles · Updated · Google Research · Oct 27
Summary
Google researchers unveiled a continual-learning “sleep framework” that lets deployed LLMs absorb new facts, preserve prior knowledge and improve reasoning instead of staying fixed after pre-training.
The method alternates active use with an offline sleep phase, where models consolidate short-term context into MoE memory blocks and run a “dreaming” loop with self-generated practice examples and reward-based refinement.
On long-context tests, the framework showed near-perfect scaling up to 10M tokens and outperformed baselines including GPT-4 and Titans by compressing context into parametric memory.
Across other benchmarks, it posted 48.9% on single-task factual incorporation, 46.2% on continued pre-training, and math scores of 79.2%, 69.0% and 46.1%, while Google said unseen-language translation saw near-zero performance drop.