confidence-sequences

Comparing sequential forecasters via confidence sequences & e-processes

让 prompt 从"感觉好了"变成"真好了" · Where "feels better" becomes "measurably better" for your AI prompts.

Taming False Positives in Out-of-Distribution Detection with Human Feedback (AISTATS '24)

Compute confidence Intervals (CLT, Chebyshev, Hoeffding) and Confidence Sequences (in a sequential setting if needed) in Julia

Official implementation of On the Tightness and Computational Tractability of Higher-Dimensional Confidence Sequences

A GxP-purpose-built LLM evaluation framework: anytime-valid validation, judge qualification, ALCOA+ audit, human plus AI workflow validation, drift monitoring, listener hooks, and a high-level facade on Inspect AI. Enables responsible AI use under risk-based assurance.

A sequential A/B test calculator using always-valid inference (mSPRT, confidence sequences), shipped with an A/A simulation showing how daily peeking inflates false positives.

Anytime valid inference in Python: confidence sequences, sequential tests and e-values

Runtime-verified fidelity for long-context LLM serving: detect, label-free, when sparse-attention KV compression silently degrades output, and bound it with anytime-valid confidence sequences. Custom HF attention backend + elastic probe scheduler. H1/H4 confirmed at 7B across Qwen2.5 & Mistral on 2x H100 · 88 tests · pre-registered hypotheses.

Sharp review-count and cost bounds with exact certified-uncertainty policies for AI-assisted financial audits

Anytime-valid sequential benchmarking of algorithms in R: confidence sequences for paired losses with a practical-equivalence margin

SOLAR: a seven-agent bot and misinformation detection framework, audited and extended into label-efficient evaluation of detectors and LLM judges. Small random human-labeled samples give valid intervals, pilot-based scorer choice and always-valid drift monitoring. Includes browser tools.

TENOR: valid evaluation of long-running agents from early surrogates and late outcomes. Reference implementation of the hanaML paper (Aug 2026).

Progressive delivery for ML models: a FastAPI gateway serving ONNX models in shadow then canary, with sequential tests deciding automatic promotion or rollback.

Progressive delivery for ML models: a FastAPI gateway serving ONNX models in shadow then canary, with sequential tests deciding automatic promotion or rollback.

Always-valid sequential A/B testing engine: mSPRT confidence sequences make peeking safe by construction, CUPED cuts variance up to 49%, SRM gates bad data. Built-in adversarial peeking harness proves the claim: naive daily peeking hit 27.5% false positives in 2,000 simulations; this engine held 1.7%. All numbers reproducible from committed seeds.

Progressive delivery for ML models: a FastAPI gateway serving ONNX models in shadow then canary, with sequential tests deciding automatic promotion or rollback.