A vLLM-compatible compression layer that expands KV cache capacity while preserving token-for-token output in named validation gates. Up to 4.2× page-level compression at 128K on the Qwen 7B page-authoritative path; 72B H200 validated at tensor-parallel = 2.
Most compression is a trade. Quantize the weights. Drop the precision. Accept the drift. The model gets smaller. The answers change.
Sfiniti AI is a different kind of compression — one that fits inside the vLLM serving stack and preserves token-for-token output in the gates we've validated. Across the model sizes and batch shapes we've tested, the same prompt produces the same answer it would have without us.
Run more inference. Preserve the output.
Page-level KV compression on the 128K page-authoritative validation gate (Qwen 7B), with token-exact output against the uncompressed baseline. 72B H200 validation at tensor-parallel = 2 retains page-level capacity.
Most public alternatives report compression with perplexity, accuracy, or quality-regression metrics. We have not found another public KV-cache compression result that combines token-level exactness against the uncompressed baseline with 70B-class tensor-parallel validation.
| Method | Compression | Output guarantee | Validated scale |
|---|---|---|---|
| Sfiniti AI | up to 4.2× page-authoritative (Qwen 7B, 128K) 2.058× multi-request vLLM gate · 1.78×–1.95× concurrency at K64/V64 (32K–128K) · up to 3.2× page-level on 72B H200 |
Token-exact (validated gates) | 7B–72B (72B at TP=2) |
| GEAR | up to 2.29× peak-memory reduction | Near-lossless (perplexity) | 7B-13B |
| TurboQuant | ~4× at 3.5 bits | Near-neutral perplexity | 70B class |
| vLLM FP8 | 2.0× | Sub-1% perplexity delta | 70B+ |
| KIVI | up to 4× (2-bit) | Quantization loss | 7B-70B |
| H2O | up to 4× | Eviction loss | 7B-13B |
Token-for-token match against the uncompressed baseline in our validated H200 gates. No drift between runs.
Qwen2.5-72B validated with tensor-parallel = 2 on NVIDIA H200, including production-style ragged batches.
Implemented as a vLLM custom backend. No retraining or fine-tune required. Production hardening in progress.
Clinical decision support and diagnostic assistance under FDA, MDR, and HIPAA frameworks. Output reproducibility is a regulatory boundary, not a nice-to-have.
Compliance review, automated underwriting, and regulated advisory. Every inference call must be reproducible for audit. Quantization-induced drift breaks the audit trail.
Contract review, eDiscovery, and any deployment under the EU AI Act. Reproducibility is not optional. Same input must produce same output, traceable to a fixed model.
For inference teams in regulated and high-fidelity workloads.