AI Inference Optimization & Low-Latency Model Serving
The model is good, but the GPU bill is climbing and responses feel slow. We build the production serving layer: engine benchmarks on your traffic, GPUs sized to real demand, and latency and cost tracked per request.
Latency lives in the serving layer, not in the model weights
vLLM, TensorRT-LLM and TGI serve the same weights with very different speed and cost. We pick engines, batching and GPU types from benchmarks on your actual prompts, then hold the result to SLOs.
What this looks like in the codebase
After the workflow is clear, we lock the technical choices that keep the product maintainable. These are the ones we use here:
- Serving engine selection benchmarked on your workload: vLLM, TensorRT-LLM or TGI against your real prompt mix
- PagedAttention-style KV-cache management and continuous batching tuned to your request sizes
- Quantization to FP8 or INT4 with speculative decoding, gated by an eval harness before rollout
- A request gateway in front of the models: routing, semantic caching and per-key rate limits
- GPU observability per request: time to first token, tokens per second and cost per request on one dashboard
Stack we ship with
Chosen because we have run it in production, not because it is fashionable.
The work inside this service
From the first data model to a production deploy. Here is what we hand over.
LLM Serving Engine Tuning
The engine decides throughput and latency more than the model does. We benchmark vLLM, TensorRT-LLM and TGI on your prompts, then tune the winner for your traffic.
- Serving engine benchmarks on your real prompt mix before any GPU is reserved
- Continuous batching and KV-cache configuration for your request sizes
- FP8 and INT4/AWQ/GPTQ quantization with eval gates that must stay green
- Speculative decoding where the latency win justifies the extra compute
GPU Cost & Capacity Engineering
Idle GPUs are where serving budgets die. We right-size instances to real throughput, scale to zero when traffic stops, and cache what repeats.
- GPU instance right-sizing from measured tokens per second, not vendor peak numbers
- Autoscaling with scale-to-zero outside working hours, spot or batch for offline jobs
- Request caching and semantic dedupe for prompts that repeat often
- Cost dashboards per team, model and endpoint with quota alerts
Model Release & Serving Operations
A new model version is a deploy with tests and rollback, not an upload. We ship changes behind canaries and only move traffic when the metrics agree.
- Blue/green and canary deployments of model versions
- A/B evaluation between model versions against your eval harness
- TTFT and tokens-per-second SLOs with alerting and error budgets
- Security controls: key management, egress rules and prompt policies
A call we made in production
Cutting p99 time to first token from 1.6s to 210ms
Cutting p99 time to first token from 1.6s to 210ms
A customer-support summarization service ran an open-weights model on default vLLM settings with reserved H100s, because nobody wanted the live endpoint competing with nightly batch jobs. p99 time to first token sat at 1.6 seconds during peak chat hours, and the GPU bill grew with every new support seat. We benchmarked vLLM against TensorRT-LLM on their real ticket mix, moved the model to FP8 after an eval replay of 400 scored summaries stayed green, and enabled prefix caching so repeated ticket threads skip recomputation. The live endpoint came down to 210ms p99 time to first token on 14 GPUs instead of 23. Batch work moved to spot instances, and GPU spend dropped 55% with eval scores unchanged.
Questions before you write to us
Ready to turn this into a product? Tell us what you need.