📊 Scoring rules are functions S(F, y) that evaluate the accuracy of a forecast distribution F, given that an outcome y was observed. PBS and PLL are strictly proper scoring rules and superior evaluation metrics for probabilistic classifiers, fixing flaws in Brier Score (MSE) and Log Loss (Cross-Entropy). Strictly proper, consistent, and better for model selection, early stopping, and checkpointing.
In many high-stakes applications, confidence calibration is critical. Traditional accuracy-based metrics (Accuracy, F1) ignore prediction confidence. Consider:
- Cancer Diagnosis: Differentiating 51% vs. 99% confidence in malignancy
- ICU Triage: Overconfident mispredictions risk patient safety
- Autonomous Vehicles: Handling uncertainties about obstacles
- Financial Risk Modeling: Pricing and investment decisions
- Security Threat Detection: High-confidence false negatives
Accuracy or F1 score alone cannot capture this nuance.
While Brier Score (Mean Squared Error, MSE, Quadratic Score) and Log Loss (Cross-Entropy, Negative Log-Likelihood, NLL, Logarithmic Score) are strictly proper scoring rules, they can still favor incorrect, overconfident predictions over more calibrated, correct ones.
| Case | True Class | Prediction | Brier Score | Log Loss | Notes |
|---|---|---|---|---|---|
| A | [0,1,0] |
[0.33,0.34,0.33] |
0.6534 | 1.0788 | ✅ Correct, but low confidence |
| B | [0,1,0] |
[0.51,0.49,0.00] |
0.5202 | 0.7133 | ❌ Incorrect, but "better" score |
Traditional scores prefer B over A, violating the principle that correct predictions should always be rewarded.
We introduce a penalty term that ensures any incorrect prediction is scored worse than any correct one.
Let y be the one‑hot true vector, p the predicted probability vector, and c the number of classes. Define the set of predictions:
Then the Penalized Brier Score (PBS) is:
And the Penalized Logarithmic Loss (PLL) is:
Penalized Brier Score (PBS)
def pbs(y, q):
"""
Computes Penalized Brier Score.
Args:
y_true: Ground truth (one-hot encoded), shape [batch_size, num_classes]
y_pred: Predicted probabilities, shape [batch_size, num_classes]
Returns:
Mean PBS across batch
"""
y = tf.cast(y, tf.float32)
c = y.get_shape()[1]
# Calculate the payoff term
ST = tf.math.subtract(q, tf.reduce_sum(tf.where(y == 1, q, y), axis=1)[:, None])
ST = tf.where(ST < 0, tf.constant(0, dtype=tf.float32), ST)
payoff = tf.reduce_sum(tf.math.ceil(ST), axis=1)
M = (c - 1) / (c)
payoff = tf.where(payoff > 0, tf.constant(M, dtype=tf.float32), payoff)
# Brier score + penalty
brier = tf.math.reduce_mean(tf.math.square(tf.math.subtract(y, q)), axis=1)
return tf.math.reduce_mean(brier + payoff)Penalized Logarithmic Loss (PLL)
def pll(y, q):
"""
Computes Penalized Logarithmic Loss.
Args:
y_true: Ground truth (one-hot encoded)
y_pred: Predicted probabilities
Returns:
Mean PLL across batch
"""
y = tf.cast(y, tf.float32)
c = y.get_shape()[1]
# Calculate the payoff term
ST = tf.math.subtract(q, tf.reduce_sum(tf.where(y == 1, q, y), axis=1)[:, None])
ST = tf.where(ST < 0, tf.constant(0, dtype=tf.float32), ST)
payoff = tf.reduce_sum(tf.math.ceil(ST), axis=1)
M = math.log(1 / c)
payoff = tf.where(payoff > 0, tf.constant(M, dtype=tf.float32), payoff)
log_loss = tf.keras.losses.categorical_crossentropy(y, q)
# Cross-entropy - penalty
ce_loss = tf.cast(log_loss, tf.float32)
return tf.math.reduce_mean(ce_loss - payoff)Install via PyPI:
pip install superior-scoring-rulesimport tensorflow as tf
from superior_scoring_rules import pbs, pll
# Sample data (batch_size=3, num_classes=4)
y_true = tf.constant([[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 0, 1]])
y_pred = tf.constant([[0.9, 0.05, 0.05, 0],
[0.1, 0.8, 0.05, 0.05],
[0.1, 0.1, 0.1, 0.7]])
print("PBS:", pbs(y_true, y_pred).numpy())
print("PLL:", pll(y_true, y_pred).numpy())class PBSCallback(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
logs = logs or {}
logs['val_pbs'] = pbs(self.validation_data[1],
self.model.predict(self.validation_data[0]))
model.fit(...,
callbacks=[
PBSCallback(),
tf.keras.callbacks.EarlyStopping(monitor='val_pbs', patience=5, mode='min'),
tf.keras.callbacks.ModelCheckpoint('best.h5', monitor='val_pbs', save_best_only=True)
]
)Below is an overview of the main files and folders:
├── Superior_Scoring_Rules.ipynb # Implementation & analysis
├── superior_scoring_rules.py # PBS & PLL functions
├── README.md # This file
├── history/ # Statistical analysis plots
└── hyperparameters-tuning/ # Tuning results
A second, independent study revisits the penalty above and asks how small it can be while still guaranteeing superiority. It derives the exact threshold in closed form (the infimal penalty), shows the penalty used above exceeds that threshold by a provable factor and is one point in a larger strictly proper family rather than a uniquely determined value, and corrects an error in the original strict-propriety proof under certain tie-breaking conventions. A pre-specified empirical case study then checks what the published penalty costs relative to the minimal one in practice -- reported together with its limitations (a non-significant effect once the experimental design's nesting is accounted for, non-replication at a higher class count, and a baseline control that turned out not to be independent of the family under test).
Code, pre-registered experiment configuration, independent verification
checks, and results for this study are in
minimal-superior-penalties/.
-
Superior scoring rules for probabilistic evaluation of single-label multi-class classification tasks
-
arXiv: 2407.17697
@article{ahmadian2025superior,
title={Superior scoring rules for probabilistic evaluation of single-label multi-class classification tasks},
author={Ahmadian, Rouhollah and Ghatee, Mehdi and Wahlstr{\"o}m, Johan},
journal={International Journal of Approximate Reasoning},
pages={109421},
year={2025},
publisher={Elsevier}
}
The follow-up study (Minimal Superior Penalties: Characterising and Calibrating Decision-Aware Proper Scoring Rules, Rouhollah Ahmadian) is currently under submission; a citation will be added here once it is available as a preprint.
This project is licensed under the BSD License.