A production-style, OpenAI-compatible proxy that caches LLM responses by meaning, not exact text — so paraphrased prompts reuse a cached answer instead of paying for a fresh, slow LLM call. Built with Python, FastAPI, sentence-transformers, and FAISS.
The hard part of semantic caching isn't matching similar prompts — it's not serving the wrong answer when two prompts look alike but mean the opposite. semcache pairs a fast bi-encoder for retrieval with an NLI cross-encoder that verifies logical agreement before returning a hit:
"Is this mushroom safe to eat?" vs "Is this mushroom NOT safe to eat?" → 0.99 cosine, but a CONTRADICTION → MISS
"What is the capital of France?" vs "France's capital city?" → paraphrase → HIT (cached answer reused)
docker run --rm -p 8000:8000 ghcr.io/brionysayani/semcache:latest
# open http://localhost:8000The published image includes both semantic models, so there is no model download
on first request. docker compose up uses the same ready-to-run image.
Or run locally:
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app # first run downloads ~370MB of modelsSend a request twice, worded differently — the second is a HIT:
curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"demo","messages":[{"role":"user","content":"What is the capital of France?"}]}'
# response carries "x_cache": "MISS"|"HIT" (also an x-cache header)The default mock backend needs no API key — clone and run. For real answers, point it at Ollama or any OpenAI-compatible provider (see below).
graph TD
Client(["Client (OpenAI SDK / curl)"]) -->|POST /v1/chat/completions| API["FastAPI Gateway"]
API --> Cache
subgraph Cache["SemanticCache (orchestrator)"]
EMB["Embedder<br/>(MiniLM bi-encoder)"]
IDX["VectorIndex<br/>(Exact NumPy | FAISS HNSW)"]
RR["ReRanker<br/>(NLI cross-encoder)"]
end
Cache -->|MISS| Backend["LLM Backend<br/>(Mock | HTTP: Ollama/OpenAI)"]
Cache -->|record| Metrics["Metrics<br/>(hit-rate, cost, latency)"]
Metrics --> Dash["Live Dashboard /stats"]
API -->|HIT: cached response| Client
A request is embedded into a vector, the index finds the nearest cached prompt, and the re-ranker verifies they truly agree. A HIT returns the cached response in milliseconds; a MISS calls the backend, stores the result, and evicts if over capacity. Every request updates the live metrics.
| Component | Role |
|---|---|
Gateway (api/) |
Validates the OpenAI-format request, runs the pipeline, returns an OpenAI-shaped body + x-cache marker. |
Embedder (providers/embedder.py) |
MiniLM bi-encoder → 384-d L2-normalized vector (so cosine = a single dot product). Pluggable. |
VectorIndex (providers/index.py) |
Nearest-neighbor search: ExactIndex (NumPy) or FaissIndex (HNSW), behind one interface. |
ReRanker (providers/reranker.py) |
NLI cross-encoder; verifies the top candidate via mutual entailment to reject contradictions. |
SemanticCache (core/cache.py) |
Orchestrates lookup → verify → store → LRU-evict; one lock keeps vector↔entry alignment safe. |
LLM Backend (providers/backend.py) |
Called only on a MISS. MockBackend ($0 default) or HTTPBackend (Ollama / any OpenAI API). |
Metrics (core/metrics.py) |
Hit-rate, cost saved, p50/p95/p99 latency → /stats JSON + live dashboard. |
The code is layered with a clean dependency direction — api → core ← providers, config at the base — so providers are swappable without touching the domain logic.
Retrieval (bi-encoder). Each prompt is embedded by MiniLM; a lookup is one matrix-vector multiply across the cache. The most similar entry is a HIT candidate if it clears sim_threshold. The bi-encoder is deliberately a loose recall gate (default 0.5) — its job is to surface candidates, not make the final call.
Verification (NLI cross-encoder). The bi-encoder captures topic, not logical agreement — it rates "is X safe" and "is X NOT safe" at ~0.99. So each candidate is verified by a Natural Language Inference model. Because NLI is directional, we require mutual entailment:
verify(a, b) = min( P_entail(a → b), P_entail(b → a) )
A paraphrase entails both ways (min stays high); a negation contradicts in at least one direction (min collapses). This is the precision filter that makes the loose retrieval gate safe.
Vector index. ExactIndex (brute-force cosine, 100% accurate, best for small N) and FaissIndex (HNSW, sub-linear, best at scale) share one interface. In auto mode (default), the cache starts exact and auto-promotes to FAISS once it crosses SEMCACHE_INDEX_PROMOTE_AT entries (default 5,000 — the benchmark crossover), preserving row ids transparently.
Eviction & accounting. Capacity-bounded with LRU eviction (behind an EvictionPolicy interface). Every MISS records token cost; every HIT records the cost it avoided (cost_saved). Latencies feed a ring buffer for percentiles.
Measured locally (Apple Silicon); reproduce with the commands in Running locally.
Correctness — labeled set of 35 prompt pairs (17 paraphrases that should hit; 18 negations / near-misses / unrelated that should miss), via eval/sweep_threshold.py:
| Pipeline | Best F1 | Precision | Negation false-hits |
|---|---|---|---|
| Bi-encoder only | 0.76 | ≤ 0.70 | ❌ all served wrong answers |
| + NLI re-ranker | 0.94 | 1.00 | ✅ all rejected |
Negations score 0.90–0.99 cosine on the bi-encoder — every one would be a false hit without verification.
Latency (scripts/demo.py, mock backend ~600ms miss): cache HIT ~100ms vs MISS ~630ms → ~6× faster (far larger against a real LLM).
Index scaling — exact vs FAISS HNSW (bench/bench_index.py, recall@1 vs exact ground truth):
| N | exact (ms) | faiss (ms) | speedup | recall@1 |
|---|---|---|---|---|
| 1,000 | 0.020 | 0.071 | 0.3× | 1.000 |
| 5,000 | 0.079 | 0.066 | 1.2× | 0.950 |
| 20,000 | 0.619 | 0.103 | 6.0× | 0.770 |
| 50,000 | 1.849 | 0.174 | 10.6× | 0.705 |
Exact wins below ~5k (no graph overhead); FAISS pulls ahead beyond it — which is why auto mode keeps both.
| Decision | Why |
|---|---|
| Bi-encoder retrieve, NLI verify | Fast topic-level retrieval + accurate logical verification on just the top candidate — speed and correctness. |
| NLI over a relevance ranker | A relevance ranker (ms-marco) scores negations as highly relevant (same topic) and fails to reject them; NLI models contradiction directly. |
| Mutual entailment | NLI is directional — a negation can entail one way. min(a→b, b→a) collapses on contradictions regardless of argument order. |
| L2-normalize at embed time | Cosine similarity becomes a plain dot product → lookup is a single matrix-vector multiply. |
| Auto-promote exact → FAISS | Run the simple exact index while it's faster; switch to ANN only past the measured ~5k crossover — no manual tuning. |
| Growable index buffer | Capacity-doubling append is amortized O(1) (naive vstack-per-add is O(N²)). |
| Mock backend default | $0, no key, reproducible for demos and CI; a real backend is one env var away. |
| Don't cache failed generations | A backend error propagates without storing — no cache poisoning with bad responses. |
| Lock the index, not the LLM call | The slow backend call runs outside the lock; only fast index mutations are guarded. |
1. The re-ranker made correctness worse. The first attempt used ms-marco-MiniLM, a relevance ranker. The eval sweep showed it lowered F1 and never improved precision — because a negation ("is X safe" vs "is X NOT safe") is highly relevant (same topic), so it scored 1.000, exactly the false hits it was meant to catch. Fix: switched to an NLI cross-encoder (nli-deberta-v3-small) that predicts contradiction/entailment directly → precision 1.00.
2. Directional NLI false hit (caught by the demo, not the eval). The eval looked perfect, but the end-to-end demo served a negation as a HIT. NLI is asymmetric: the eval happened to pass pairs as (stored, query) where the negation scored 0.002, but the live path passes (query, stored), where the same pair scored 0.919 and was accepted. Fix: require mutual entailment — min(entail(a,b), entail(b,a)) — plus a regression test asserting rejection in both argument orders.
3. O(N²) index append hung the benchmark. Building a large corpus froze at 94% CPU because ExactIndex.add() did np.vstack([...]) per call, reallocating the whole matrix each time. Fix: a growable buffer (capacity doubling) → amortized O(1) append; search operates on the live [:n] view.
Served at / (polls /stats every 2s):
- Try it — type a prompt, send it (
⌘/Ctrl+Enter), and see the live HIT/MISS badge, latency, token count, and response. One-click example chips (including the mushroom negation pair) demo hits and the re-ranker rejection in real time. - Hit rate, cost saved vs spent, latency (hit/miss p50, p99), and cache size / capacity / active index type.
| Method | Endpoint | Description |
|---|---|---|
POST |
/v1/chat/completions |
OpenAI-compatible chat proxy (semantic-cached); returns an OpenAI body + x_cache. |
GET |
/stats |
Metrics JSON. |
GET |
/ |
Live dashboard. |
GET |
/health |
Liveness probe. |
semcache/
├── app/ # layered: api → core ← providers, config at the base
│ ├── main.py # app factory + lifespan (injects providers into the core)
│ ├── api/ # routes.py (HTTP handlers) + schemas.py (OpenAI wire models)
│ ├── core/ # cache.py, metrics.py, models.py, tokens.py — no framework deps
│ ├── providers/ # embedder, index (Exact/Faiss/Auto), reranker, backend, eviction
│ └── config/ # settings.py (SEMCACHE_* env) + constants.py
├── static/stats.html # live dashboard
├── eval/ # labeled dataset + precision/recall/F1 threshold sweep
├── bench/ # exact-vs-FAISS latency/recall benchmark
├── scripts/demo.py # end-to-end showcase
├── tests/ # cache, index, api, reranker (pytest)
├── Dockerfile · docker-compose.yml · .github/workflows/ci.yml · .env.example
└── DESIGN.md # design doc: goals, architecture, decisions, trade-offs
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m scripts.demo # end-to-end showcase (hits, negation rejected, savings)
uvicorn app.main:app # gateway + dashboard at http://localhost:8000
# reproduce the numbers
python -m eval.sweep_threshold --rerank # precision/recall/F1 (precision → 1.00)
python -m bench.bench_index # exact vs FAISS → bench/results/
# tests
pytest -m "not model" # fast unit/API suite — what CI runs
KMP_DUPLICATE_LIB_OK=TRUE pytest # full suite incl. real-model reranker testsmacOS note: faiss and torch each bundle their own OpenMP runtime; loading both in one process aborts unless
KMP_DUPLICATE_LIB_OK=TRUEis set. CI runs the fast lane (Linux), which is unaffected.
The default mock backend returns placeholder text. For real answers, point semcache at a local Ollama server — same HTTPBackend, no code change:
ollama serve & ollama pull llama3.2:1b
SEMCACHE_BACKEND=http SEMCACHE_HTTP_MODEL=llama3.2:1b uvicorn app.main:appA MISS now calls Ollama for a real answer; a paraphrase HIT serves that same answer from cache with no second call. The same backend works against any OpenAI-compatible provider via SEMCACHE_HTTP_BASE_URL / SEMCACHE_HTTP_API_KEY. All settings are env-overridable — see .env.example.
In-memory (lost on restart; production would persist vectors + responses to a vector DB + Redis) · matches on the last user message only (ignores multi-turn context) · single-process (no shared cache across instances) · NLI adds latency on hits (toggle via SEMCACHE_RERANK_ENABLED) · plain-text logging (production would emit structured/JSON logs with a per-request correlation id for aggregation and tracing).
Apache License 2.0 — see LICENSE.