Operating an AI search engine that handles roughly 400 million queries per month carries significant costs. Perplexity AI has been quietly constructing infrastructure to reduce those expenses, publishing research on custom serving technology that ranges from compact embedding models to a proprietary engine capable of running trillion-parameter models.

The effort reflects a broader industry reality: AI search engines absorb a compute cost with every query that traditional keyword search does not, which is why inference efficiency has become a central battleground for companies racing to make AI answers economical at consumer scale.

Compact embedding models for retrieval

Central to Perplexity's retrieval improvements is the pplx-embed family of models, comprising pplx-embed-v1 and pplx-embed-context-v1. The models come in two sizes: a 0.6 billion parameter variant and a larger 4 billion parameter version. Both were released in February 2026.

Using quantization techniques — which reduce the numerical precision of model weights without destroying accuracy — Perplexity claims storage requirement reductions of up to 32x. Both models were optimized specifically for low-storage deployment, a design choice reflecting the practical realities of web-scale retrieval, where every query requires searching vast indexes of embedded documents.

ROSE and custom networking

The more ambitious piece of infrastructure is ROSE, short for Runtime-Optimized Serving Engine. Introduced around February 2025, ROSE is built to serve a wide range of model architectures, including the increasingly popular Mixture-of-Experts (MoE) format that underpins many of today's largest language models. MoE architectures activate only a fraction of a model's parameters per query, but they place heavy demands on inter-GPU communication, which is why networking efficiency is a key lever for serving them affordably.

Perplexity tackled MoE serving with custom RDMA networking optimizations achieving up to 97.1% bandwidth efficiency. RDMA, or Remote Direct Memory Access, allows GPUs to communicate without going through the CPU. The engine runs across AWS Elastic Fabric Adapter environments. In November 2025, Perplexity published its first paper on arXiv detailing these optimizations and open-sourced a library called fabric-lib, containing custom MoE kernels that the company says deliver state-of-the-art latencies for trillion-parameter model serving. The decision to open-source the work aligns Perplexity with a wider trend of AI labs releasing serving infrastructure research publicly, as was the case with DeepSeek's disclosure of its inference optimization techniques.

Hybrid inference and on-device processing

In June 2026, Perplexity unveiled a hybrid local-cloud inference orchestrator. The system automatically routes tasks between on-device models running on local hardware and more powerful cloud-based models, depending on query complexity. The orchestrator supports multiple local silicon options, and keeping certain queries on-device means they never touch a server — eliminating their serving cost entirely while also addressing latency and privacy considerations.

Position in the inference arms race

Perplexity runs on NVIDIA infrastructure and maintains strategic partnerships with both NVIDIA and CoreWeave, the GPU cloud provider that has become a go-to for AI workloads. At 400 million monthly queries, ROSE's 97.1% bandwidth efficiency and the 32x storage reduction from quantized embeddings translate directly into infrastructure cost savings. What bears watching going forward is whether Perplexity's published serving stack and hybrid orchestrator become differentiators against larger rivals that also build their own inference infrastructure, and whether the open-sourced fabric-lib gains adoption across the broader AI serving community.