Concluded·Day 4·6 sources·updated 4 Sep 2026

Perplexity's Lily local inference engine

The gist

Perplexity open-sources Lily, 1.35x faster than MLX-LM on Apple Silicon

Lily shows that an inference engine constrained to one model on one hardware family can outperform a general-purpose framework by a measurable margin while maintaining near-identical output quality. The open-source release makes the approach and its engineering trade-offs available for inspection and reuse.

The full picture

Lily, an open-source inference engine from Perplexity, is available on GitHub for running Qwen3.6-35B-A3B on Apple Silicon. It serves as the local component of Hybrid Compute in Perplexity Computer, the company's Mac application, enabling on-device processing alongside cloud-based computation. Lily is built in Rust with hand-written Metal kernels and no PyTorch or MLX in the execution path. On an Apple M5 Max with 128 GB RAM, it averaged 4,156 prefill tokens/s versus MLX-LM's 3,388 and 170.0 decode tokens/s versus 126.4. A teacher-forced correctness check across 192 token positions found Lily's perplexity only 0.04% higher than the reference model, with the top-ranked token identical 96.35% of the time. Speculative decoding was tested and found to make inference 18% slower. Lily exposes an OpenAI-compatible chat-completions API that streams tokens.

How it developed

2 September 2026

Perplexity open-sourced Lily on September 2, a Rust-based inference engine with hand-written Metal kernels, no PyTorch or MLX in the execution path, built to run Qwen3.6-35B-A3B on Apple Silicon as the local component of Hybrid Compute in Perplexity Computer, the company's Mac application.

On an M5 Max with 128 GB RAM, Lily averaged 4,156 prefill tokens/s versus MLX-LM's 3,388 and 170.0 decode tokens/s versus 126.4, with a correctness check finding its perplexity only 0.04% higher than the reference model and top-ranked token identical 96.35% of the time. Speculative decoding was tested and found to slow inference by 18%.

First citedPerplexity Twitter The Neuron tech-insider.org explainx.ai+2

1 September 2026

Perplexity introduces hybrid compute in Perplexity Computer, enabling tasks to shift from cloud to local Mac model for sensitive data

Sources

Perplexity Twitter