Forecast verification
- The philosophy of forecast verification is rather different between deterministic and probabilistic verification metrics: generally speaking, deterministic metrics measure differences, whereas probabilistic metrics assess reliability and sharpness of predictive distributions. This article considers the root-mean-square error (RMSE), which can be seen as a deterministic metric, and the probabilistic metric Continuous Ranked Probability Score (CRPS), and demonstrates that under certain conditions, the CRPS can be mathematically expressed in terms of the RMSE when these metrics are aggregated. One of the required conditions is the normality of distributions. The other condition is that, while the forecast ensemble need not be calibrated, any bias or over/underdispersion cannot depend on the forecast distribution itself. Under these conditions, the CRPS is a fraction of the RMSE, and this fraction depends only on the heteroscedasticity of the ensemble spread and the measures of calibration. The derived CRPS-RMSE relationship for the caseThe philosophy of forecast verification is rather different between deterministic and probabilistic verification metrics: generally speaking, deterministic metrics measure differences, whereas probabilistic metrics assess reliability and sharpness of predictive distributions. This article considers the root-mean-square error (RMSE), which can be seen as a deterministic metric, and the probabilistic metric Continuous Ranked Probability Score (CRPS), and demonstrates that under certain conditions, the CRPS can be mathematically expressed in terms of the RMSE when these metrics are aggregated. One of the required conditions is the normality of distributions. The other condition is that, while the forecast ensemble need not be calibrated, any bias or over/underdispersion cannot depend on the forecast distribution itself. Under these conditions, the CRPS is a fraction of the RMSE, and this fraction depends only on the heteroscedasticity of the ensemble spread and the measures of calibration. The derived CRPS-RMSE relationship for the case of perfect ensemble reliability is tested on simulations of idealised two-dimensional barotropic turbulence. Results suggest that the relationship holds approximately despite the normality condition not being met.…
Author details: | Tsz Yan LeungORCiD, Martin LeutbecherORCiDGND, Sebastian ReichORCiDGND, Theodore G. ShepherdORCiDGND |
---|---|
DOI: | https://doi.org/10.1002/qj.4120 |
ISSN: | 0035-9009 |
ISSN: | 1477-870X |
Title of parent work (English): | Quarterly journal of the Royal Meteorological Society |
Subtitle (English): | relating deterministic and probabilistic metrics |
Publisher: | Wiley |
Place of publishing: | Hoboken |
Publication type: | Article |
Language: | English |
Date of first publication: | 2021/07/07 |
Publication year: | 2021 |
Release date: | 2024/01/23 |
Tag: | CRPS; NWP; RMSE; ensembles; idealised turbulence; verification |
Volume: | 147 |
Issue: | 739 |
Number of pages: | 11 |
First page: | 3124 |
Last Page: | 3134 |
Funding institution: | U.K. Engineering and Physical Sciences Research CouncilUK Research & Innovation (UKRI)Engineering & Physical Sciences Research Council (EPSRC) [EP/L016613/1]; European Research CouncilEuropean Research Council (ERC)European Commission [339390]; Deutsche ForschungsgemeinschaftGerman Research Foundation (DFG) [SFB 1114/2/235221301] |
Organizational units: | Mathematisch-Naturwissenschaftliche Fakultät / Institut für Mathematik |
DDC classification: | 5 Naturwissenschaften und Mathematik / 51 Mathematik / 510 Mathematik |
Peer review: | Referiert |
Publishing method: | Open Access / Hybrid Open-Access |
License (German): | CC-BY - Namensnennung 4.0 International |