[alejandroschuler]

Closes #352.

Cause

The overflow warnings in Normal (overflow encountered in exp at self.scale = np.exp(params[1]) and overflow encountered in square at self.var = self.scale**2) come from trial steps that the line search rejects.

The line search tests start - scale * resids for scale 1, 2, ..., 512. These trial steps are not multiplied by the learning rate, so each one is 1/learning_rate times the step that the fit applies. If a trial step overflows, its loss is not finite, and the line search rejects it and halves the scale. The model does not change. Only the warnings get through.

The most common way to get these warnings is plain gradients on a target with large units. With natural_gradient=False and CRPScore, the gradient for log sigma is sigma * (2 phi(Z) - 1/sqrt(pi)), so its size grows with the units of Y. When the predicted standard deviation is above about 600, the first trial step (scale 1) puts log sigma above 355, and scale**2 overflows. This reproduces the issue on master:

rng = np.random.default_rng(0)
X = rng.normal(size=(500, 3))
y = 1000 * (X[:, 0] + rng.normal(size=500))
NGBRegressor(Dist=Normal, Score=CRPScore, natural_gradient=False, n_estimators=20).fit(X, y)

On master this fit gives 94 overflow warnings, and all predictions are finite. Laplace and Exponential with CRPScore and Poisson with LogScore overflow in the same way with plain gradients.

Deep trees with min_samples_leaf=1 cause a different problem. They fit each training point exactly, and the predicted scale collapses toward 0 on the training data. The collapse also makes trial steps overflow, and early stopping ends the fit before the collapse. This is probably why early stopping removed the warnings for one reporter. At a high learning rate (0.5 or more), the collapse can make a real parameter overflow. The gradient then has NaN values, and the base learner fails with Input y contains NaN, as in #61.

Changes

• NGBoost.line_search computes the trial losses inside np.errstate(over="ignore", divide="ignore", invalid="ignore"). The line search already rejects a non-finite loss, so the fitted model does not change. The loss at the current parameters stays outside the errstate, so real overflows still warn.
• partial_fit checks the gradient before it fits the base learners. If the gradient has non-finite values, it raises a ValueError that says the fit diverged at iteration k and gives settings to try. For scikit-learn learners this replaces Input y contains NaN at the same iteration, with the same exception type. A base learner that accepts NaN targets now gets this error and is not given a bad target.
• The tuning page of the user guide has a new section, "Overflow Warnings and Diverging Fits". It says that plain gradients depend on the units of Y (standardize Y, or keep natural_gradient=True). It also says that deep trees with very small leaves collapse the predicted scale, and that shallower trees, a lower learning rate or early stopping prevent this. The new cells are markdown only, between "Early Stopping" and "Using sklearn Model Selection", so they do not touch the cells that Run the user guide notebooks in the docs build #431 changes.
• Release notes.

Evidence

The same fits with the master line search and with this branch:

case	RuntimeWarnings, master -> this PR	params bit-identical
Normal, plain CRPS, sd(Y) ~ 1000	32 -> 0	yes
Normal, natural CRPS, deep trees, diabetes	8 -> 0	yes
Laplace, plain CRPS, sd(Y) ~ 1000	0 -> 0	yes
Exponential, plain CRPS	296 -> 0	yes
Poisson, plain LogScore	1243 -> 0	yes
control: Normal, natural LogScore, California housing	0 -> 0	yes
control: LogNormal, natural CRPS, California housing	0 -> 0	yes

Normal with natural LogScore, deep trees and learning_rate=0.5 on the diabetes data now stops with The fit diverged at iteration 239: ... and not with Input y contains NaN.

The guide's claim about plain gradients: with Normal, CRPScore, sd(Y) about 1000 and 500 trees, the predicted mean has a standard deviation of 0.01 with natural_gradient=False and 858 with natural_gradient=True. The standard deviation of the true signal is 872.

Tests

tests/test_numerics.py:

• test_rejected_line_search_steps_do_not_warn fits the reproducer above with RuntimeWarning set to raise an error. It fails on master.
• test_fit_raises_clear_error_when_gradient_is_not_finite puts a NaN into the Normal gradient and checks for the new error.

I ran the full test suite, black, isort, flake8 and pylint (10.00/10).

Not in this PR

• I did not clip the log scale in Normal. Clipping also removes the warnings, but it changes fits in pathological runs. Today the overflowing trial steps are rejected. With a clip they are finite and can be accepted, and in one test with learning_rate=1.0 the test CRPS went from 63.5 to 5279.
• NGBRegressor(Dist=T) with default settings fails with Input y contains NaN at about iteration 25, at every scale of Y that I tried. This is a separate bug in the T distribution. With this PR it gets the new "diverged" error.

🤖 Generated with Claude Code