Perch · multi-model LLM serving
More models per GPU. No lost answers.
Perch runs beneath unmodified vLLM and lets each GPU serve many models while holding first-token and per-token deadlines. Your clients keep the same OpenAI-compatible API.
Evaluation release 0.1.1 · unmodified vLLM 0.31 · 2× H200 NVL · every figure measured
Qwen2.5-7B models on an 80 GB-class card. Perch served 10 per GPU at 98.1% of tokens on time, with no answers cut off.
What we are building
One new layer. Nothing else changes.
Perch sits between your clients and unmodified vLLM. Clients keep calling the same OpenAI-compatible API, vLLM keeps running your models, and each GPU serves many more of them while first-token and per-token deadlines hold.
vLLM as shippedOne model per GPU
- Your clientsOpenAI-compatible API
- A routerOne endpoint per model
- vLLMOne engine, one model
- GPUMemory held by one model
Each model keeps a GPU whether it is busy or not. Long-tail catalogs pay for idle cards.
With PerchOne new layer
- Your clientsOpenAI API, unchanged
- PerchMany models, deadlines held
- vLLMUnmodified
- GPUsMany models per card
Each GPU serves many more models, first-token and per-token deadlines hold, and no answer is lost.
What you get
Fewer GPUs for the same catalog.
Long-tail catalogs leave most models idle most of the time, yet each one holds a GPU. Perch lets those models share GPUs without giving up latency or answers.
10 per GPU
Where plain colocation fits 3 to 4 7B models on an 80 GB-class card, Perch served 10.
Held, as you set them
You set the first-token and per-token targets; Perch plans every GPU around them.
Nothing cut off
Every answer completes, and it is the same answer the model gives when it runs alone.
Same API, same vLLM
Clients keep the OpenAI-compatible API; vLLM stays unmodified; your models stay as they are.
Results
Measured on two H200 GPUs.
2× NVIDIA H200 NVL, vLLM 0.31, Qwen2.5-7B models, a 4 s first-token and 100 ms per-token deadline, bursty long-tail traffic.
| Setup | Tokens on time | Answers lost | GPUs vs one model per GPU |
|---|---|---|---|
| Perch, 6 models on 2 GPUs | 99.9–100% | 0 | 3× fewer |
| Perch, 20 models on 2 GPUs (10 per GPU) | 98.1% | 0 cut off, 4 of 752 timed out | 10× fewer |
| Stock vLLM sharing the GPUs, 6 models on 2 GPUs | 84.1% | 183 of 840 cut off | – |
measured Same traffic for every row; the stock row uses vLLM's own built-in way of sharing a GPU between models. Outputs with Perch are identical to each model served alone. With the memory budget of an 80 GB-class card, plain colocation fits 3 to 4 such models per GPU; Perch served 10.
The release
Four parts, one per job.
Linux binaries and compiled wheels for x86_64 and aarch64 (Grace). Perch is proprietary software: free to evaluate, licensed for pilots and production.
perch-agentOne per GPU node
The OpenAI-compatible endpoint your clients call, with status and Prometheus metrics.
perch-engineFor vLLM
A compiled vLLM plugin. No fork of vLLM.
perch-gatewayAcross nodes
Routes clients to the right node, from a static list or Kubernetes pods.
perch-simBefore you buy GPUs
Estimates from your own request log how many of your models fit per GPU at your deadlines.
Licensing
Evaluate free. Key for pilots and production.
Perch runs without a key in evaluation mode, large enough to measure the gain on your own models. A signed licence key from ZenEdgeX lifts the limits for a pilot or for production. Your models, prompts and traffic stay yours: Perch sends nothing to ZenEdgeX.
2 GPUs, 4 models per GPU
Per host, on hardware you control, to measure Perch with your own models and traffic. Not for production traffic.
Limits lifted, for a pilot
A signed key for a defined pilot on your fleet, under a pilot agreement with ZenEdgeX.
Serving your customers
Production and commercial use need a production key, under a licence agreement with ZenEdgeX.
# install a key for the user that runs vLLM and the agent mkdir -p ~/.perch && cp acme.license ~/.perch/license # or PERCH_LICENSE=<key> perch-agent --check-licence # /status also shows the active licence
The installer asks you to accept the evaluation licence before it installs anything. Pilot and production keys: hello@zenedgex.in.
Try it
Fifteen minutes on one GPU node.
Download the bundle for your CPU from the latest release, install it on each GPU node, and point your clients at Perch. The install guide covers starting your vLLM engines and the agent configuration.
# install (per GPU node) tar xzf perch-0.1.1-linux-x86_64.tar.gz && cd perch-0.1.1 sudo PYTHON=$(which python3) ./install.sh # asks you to accept the licence # start Perch, then call it like any OpenAI endpoint perch-agent --config agent.toml curl localhost:8080/v1/completions -H 'content-type: application/json' \ -d '{"model": "qwen-7b", "prompt": "Hello", "max_tokens": 32}'
| You need | |
|---|---|
| OS / CPU | Linux, x86_64 or aarch64 (Grace) |
| GPU | NVIDIA, Ampere or newer |
| Engine | vLLM 0.31, Python 3.10 to 3.12 |
| Without a key | Up to 2 GPUs and 4 models per GPU per host |
Full guide: INSTALL.md. Checksums are in SHA256SUMS on the release.
Status
Where Perch stands today.
In plain language, for teams deciding whether to try it.
Many models per GPU
10 Qwen2.5-7B models per H200 at 98.1% of tokens on time, no answers cut off.
Evaluation release
Version 0.1.1 for Linux x86_64 and aarch64: the agent, plugin, gateway and simulator, with licence keys for pilots and production.
Design partners
Providers serving many fine-tuned or long-tail models: send a two-week request log and get a capacity report for your catalog.
Today's scope
NVIDIA GPUs and vLLM only. perch-sim estimates run 2 to 3 points optimistic; plan with that margin.
Your catalog
Send a request log, get a capacity report.
A two-week log of model id, timestamp, and prompt and output lengths, with no content. We return how many of your models fit per GPU at your deadlines, and how many GPUs you need.
hello@zenedgex.in