TY  - JOUR
A1  - Leung, Tsz Yan
A1  - Leutbecher, Martin
A1  - Reich, Sebastian
A1  - Shepherd, Theodore G.
T1  - Forecast verification
BT  - relating deterministic and probabilistic metrics
JF  - Quarterly journal of the Royal Meteorological Society
N2  - The philosophy of forecast verification is rather different between deterministic and probabilistic verification metrics: generally speaking, deterministic metrics measure differences, whereas probabilistic metrics assess reliability and sharpness of predictive distributions. This article considers the root-mean-square error (RMSE), which can be seen as a deterministic metric, and the probabilistic metric Continuous Ranked Probability Score (CRPS), and demonstrates that under certain conditions, the CRPS can be mathematically expressed in terms of the RMSE when these metrics are aggregated. One of the required conditions is the normality of distributions. The other condition is that, while the forecast ensemble need not be calibrated, any bias or over/underdispersion cannot depend on the forecast distribution itself. Under these conditions, the CRPS is a fraction of the RMSE, and this fraction depends only on the heteroscedasticity of the ensemble spread and the measures of calibration. The derived CRPS-RMSE relationship for the case of perfect ensemble reliability is tested on simulations of idealised two-dimensional barotropic turbulence. Results suggest that the relationship holds approximately despite the normality condition not being met.
KW  - CRPS
KW  - ensembles
KW  - idealised turbulence
KW  - NWP
KW  - RMSE
KW  - verification
Y1  - 2021
U6  - https://doi.org/10.1002/qj.4120
SN  - 0035-9009
SN  - 1477-870X
VL  - 147
IS  - 739
SP  - 3124
EP  - 3134
PB  - Wiley
CY  - Hoboken
ER  - 
TY  - JOUR
A1  - Lorenz, Claas
A1  - Clemens, Vera Elisabeth
A1  - Schrötter, Max
A1  - Schnor, Bettina
T1  - Continuous verification of network security compliance
JF  - IEEE transactions on network and service management
N2  - Continuous verification of network security compliance is an accepted need. Especially, the analysis of stateful packet filters plays a central role for network security in practice. But the few existing tools which support the analysis of stateful packet filters are based on general applicable formal methods like Satifiability Modulo Theories (SMT) or theorem prover and show runtimes in the order of minutes to hours making them unsuitable for continuous compliance verification. In this work, we address these challenges and present the concept of state shell interweaving to transform a stateful firewall rule set into a stateless rule set. This allows us to reuse any fast domain specific engine from the field of data plane verification tools leveraging smart, very fast, and domain specialized data structures and algorithms including Header Space Analysis (HSA). First, we introduce the formal language FPL that enables a high-level human-understandable specification of the desired state of network security. Second, we demonstrate the instantiation of a compliance process using a verification framework that analyzes the configuration of complex networks and devices - including stateful firewalls - for compliance with FPL policies. Our evaluation results show the scalability of the presented approach for the well known Internet2 and Stanford benchmarks as well as for large firewall rule sets where it outscales state-of-the-art tools by a factor of over 41.
KW  - Security
KW  - Tools
KW  - Network security
KW  - Engines
KW  - Benchmark testing;
KW  - Analytical models
KW  - Scalability
KW  - Network
KW  - security
KW  - compliance
KW  - formal
KW  - verification
Y1  - 2021
U6  - https://doi.org/10.1109/TNSM.2021.3130290
SN  - 1932-4537
VL  - 19
IS  - 2
SP  - 1729
EP  - 1745
PB  - Institute of Electrical and Electronics Engineers
CY  - New York
ER  -