Google Speeds Up Complex AI Search by 20x
Google Research unveiled Retrieve-for-Train, a framework that bypasses heavy LLM inference bottlenecks to accelerate complex, multi-query AI searches by up to 20 times.
Google researchers Pengcheng Jiang and Judith Yue Li have developed Retrieve-for-Train, a framework designed to eliminate the latency bottlenecks of complex AI search. Traditional search systems often perform query fan-out, breaking a single prompt into multiple sub-queries to return a diverse set of results. However, using large language models to decompose queries dynamically at inference time requires a massive thinking budget and generates hundreds of sequential chain-of-thought tokens, slowing down response times.
To solve this, the framework shifts the heavy computational work to an offline training phase. First, researchers use reinforcement learning to train a fan-out language model, specifically utilizing 4B open-source models like Gemma3-4B and Qwen3-4B. This training employs Soft-GRPO, which combines group relative policy optimization with soft proximal policy optimization. The models are optimized using a composite reward that balances groundedness, alignment, and diversity, the latter measured via the Vendi Score. The trained model then generates exactly 10 sub-queries per prompt to synthesize supervision data. Finally, this data trains a compact, 53.9-million-parameter diffusion retriever that maps query embeddings directly to target embeddings in a single, non-autoregressive pass.
The researchers evaluated the system using a CLIP-based retriever on a large-scale fashion dataset and the MuLan retriever on a proprietary music playlist dataset. The distilled 53.9M-parameter diffusion model achieved a 12x to 20x speedup over traditional autoregressive methods. While standard autoregressive fan-out latency can stretch to nearly 50 seconds under large context batches, the Retrieve-for-Train diffusion model keeps latency between sub-second and a few seconds.
For search and recommendation system practitioners, this development makes high-quality, diverse set-based retrieval viable for real-time production environments. By acting as what the researchers call a "one-time objective transducer" rather than an online inference engine, the framework allows developers to deploy lightweight models that deliver expert-level, diverse search results instantly, bypassing the high computational costs and latency of online LLM reasoning.
This is our own summary of reporting by Google Research
