NICE metrics: evaluating forecasts against persistence

Cyril Voyant · Updated 2026-10-18

Why a reference matters

An absolute error describes forecast quality on a given series. A reference-relative error also asks whether a model improves over a specified baseline on exactly the same observations.

For forecast errors e and persistence errors p, the NICE component of order k is the ratio of their L-k norms:

NICE_k = (sum(abs(e)^k) / sum(abs(p)^k))^(1/k)

For the same number of samples this also equals the ratio of the corresponding mean-power errors. Orders 1, 2 and 3 place progressively more weight on large deviations.

Interpretation

• Below 1: smaller error norm than the selected persistence baseline.
• Equal to 1: the same norm as that baseline.
• Above 1: worse than the baseline on that component.

Do not confuse the error ratio with a skill score defined as one minus the ratio. State the exact formula whenever an aggregated NICE quantity or transformed score is used.

Small reproducible example

With errors [1, 2] and persistence errors [2, 4], every component is 0.5. If the persistence error norm is zero, the ratio is undefined: report that case rather than silently adding an arbitrary constant.

Reporting protocol

State persistence definition, horizon, temporal alignment, scoring window and missing-value mask. Report absolute errors alongside the ratio. A ratio below one does not establish operational value or superiority to every possible baseline.

Reference and implementation

NICEᵏ Metrics: Unified and Multidimensional Framework for Evaluating Deterministic Solar Forecasting Accuracy, 2025. Credit all authors as listed in the publication. The PV_ELM_pred repository provides the metric implementation and benchmark conventions. See PV forecasting protocol.