Integrating SGLang into Spyre in Under a Week with AI

TL;DR

With help from AI coding agents, our two-person team integrated SGLang with IBM Spyre in less than a week, building a plugin of roughly 1,500 lines. Existing Spyre support in PyTorch and SGLang’s clear extension points made it straightforward to reuse the hardware code. SGLang delivered about 3.6× the throughput of the reference PyTorch inference implementation using torch-spyre. It was also on par with vLLM on the workloads measured for Granite 3.3 8B and Gemma 4 26B A4B. Prefix sharing through RadixCache worked without any Spyre-specific changes.

We work on IBM’s Spyre accelerator and wanted to make SGLang‘s serving stack, including its built-in prefix sharing, available to Spyre users.

In less than a week, the two of us used AI-assisted development to integrate SGLang into IBM Spyre with a plugin of just ~1,500 lines of code. A key factor was building on existing PyTorch hardware support and SGLang’s well-defined extension points. This allowed the two of us, alongside our AI coding agents, to integrate a new accelerator into SGLang in a remarkably short time. The AI coding agents used existing PyTorch implementations of Spyre operations as reference code and adapted the integration code to SGLang’s backend interfaces. Their work focused on connecting the existing hardware support to the new serving framework.

Spyre and the starting point

Spyre is IBM’s accelerator for enterprise AI inference. It integrates using torch-spyre, which registers Spyre as a PyTorch device and connects it to torch.compile, exposing hardware support through standard PyTorch APIs.

The result

The initial SGLang integration took the two of us less than a week of work. The plugin was roughly 1,500 lines, building on the compiler and reusable hardware support from PyTorch. Four small patches extended SGLang’s backend support: two made existing execution paths available without CUDA-specific dependencies, and two registered Spyre as a supported device and attention backend that does not use Triton. The compatibility changes can also support other accelerators; the registration changes are specific to Spyre.

SGLang on Spyre achieved approximately 3.6× the throughput of the reference PyTorch inference implementation using torch-spyre.

We also compared SGLang and vLLM on Spyre using Granite 3.3 8B and Gemma 4 26B A4B. Across the workloads we measured for these two models, SGLang’s performance was on par with vLLM.

Relative performance of SGLang and vLLM / spyre-inference on Granite 3.3 8B and Gemma 4 26B-A4B, normalized to vLLM = 1.0. Lower is better.

Why it worked

With a working compiler and an existing inference server in place, adding SGLang became a focused integration task. PyTorch and torch.compile supplied a shared implementation layer, expressing accelerator functionality through APIs that both serving frameworks could use.

SGLang supplied clear places to connect that code. Its platform registration, attention backend, KV pool, and operator dispatch separated the integration into bounded tasks while preserving the existing scheduler, request handling, and serving loop. The reference code came from the existing vLLM integration for Spyre, including its PyTorch implementations of accelerator operations. This gave the AI coding agents concrete code for Spyre’s hardware behavior, while SGLang’s extension points defined the interfaces their adapted code had to satisfy. Together, these reduced how much the agents had to infer about the hardware and where the integration belonged.

The existing vLLM Spyre backend was about 3,500 lines, compared with roughly 1,500 for the SGLang plugin. In the initial integration, the attention base class supplied the surrounding logic, leaving the Spyre backend to override the inner scaled dot-product attention operation. Reusing that base class accounted for roughly two-thirds of the line-count difference between the two plugins. SGLang’s device-agnostic KV pool also allowed a single method override for buffer creation; allocation still needed PyTorch-based handling where the existing path relied on Triton.

Reusing the vLLM Spyre implementation

We reused the attention code from the vLLM Spyre integration, including how it groups inputs into buckets and handles tensor shapes. We also reused its PyTorch-based approach to allocating the KV cache. For RMSNorm, SiLU, GeLU, rotary embeddings, and linear layers, we connected the existing Spyre implementations to SGLang’s operator dispatch.

Some features required no Spyre-specific work at all. RadixCache is a good example: prefix sharing is resolved entirely within SGLang’s scheduler before execution reaches the accelerator, so the existing scheduling logic worked unchanged.

The plugin is available as open source alongside upstream SGLang.

Using SGLang with Spyre

The example assumes the Spyre runtime, torch-spyre, the SGLang plugin, and its compatibility patches are installed and configured.

import sglang as sgl models = [ "ibm-ai-platform/micro-g3.3-8b-instruct-1b", "mistralai/Ministral-8B-Instruct-2410", ] for model in models: engine = sgl.Engine( model_path=model, device="spyre", attention_backend="spyre_paged", dtype="float16", page_size=64, max_running_requests=1, disable_cuda_graph=True, ) try: output = engine.generate( "Why is the sky blue?", {"temperature": 0, "max_new_tokens": 64}, ) print(model, output["text"]) finally: engine.shutdown()

Looking ahead

The Spyre software stack continues to evolve rapidly.

Over the coming months, we expect to add support for dynamic shapes, reducing much of the shape management required today. Later this year, we also expect to bring Triton support to Spyre and enable RadixAttention on top of it, opening the door to additional kernel optimizations.

Keeping hardware support in PyTorch and integration code at SGLang’s extension points gives future compiler and runtime improvements a path into the serving stack. The next integration can start from the same shared foundation, with concrete implementations and interfaces that developers and AI coding tools can work with.

The post Integrating SGLang into Spyre in Under a Week with AI appeared first on PyTorch.