Open Source
Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting
arXiv stat.ML0 views
arXiv:2602.10182v2 Announce Type: replace-cross Abstract: Standard sample-based metrics like Continuous Ranked Probability Score (CRPS) and Quantile Loss (QL) are frequently used in the evaluation of probabilistic time-series forecasting. However, these metrics fail to capture multivariate correlations or assess the ability to capture distribution tails. These global metrics are insensitive to errors on high-utility events in distributional tails due to over-representation of the distribution's body. To accurately measure distributional fidelity on tail-regions and regions of high utility while still having a proper scoring rule, we introduce a family of censored signature kernel metrics. Our proposed metrics concentrate evaluation of forecasts to a focus region, representing high-utility or distributional tails, by collapsing the body of the distribution to a single pivot. Our benchmarks on time-series foundation models (TSFMs) reveal that while a clear ranking can be formed for distribution capture, there is no clear winner for tasks like systemic load prediction. This indicates that models with a strong ability to capture an overall distribution do not produce forecasts with high downstream utility. To encourage the use of the proposed metrics we open-source the efficient signature kernel (ESK) library. This library facilitates batch computation, with custom triton kernels, achieving a speed-up of up to 3.58$\times$ compared to implementations via the popular SigKernel library. Code for our metrics are available on GitHub.
Source:https://arxiv.org/abs/2602.10182PreviousPragmatic Curiosity: A Unified Framework for Hybrid Learning and Optimization via Active InferenceNextKinetic Langevin Splitting Schemes for Constrained Sampling
Related posts
YUL: Bringing Software Supply Chain Security to Coding Agents
0arXiv cs.SE
Towards Pose-Robust Few-Shot Vision-Language-Action Models
0arXiv cs.RO
High-Precision Docking of Differential-Drive Mobile Robots Through LiDAR and Vision Collaboration
0arXiv cs.RO
Building Your Own: A Practical Guide to Datasets, Pipelines and Evaluation for an AI Intrusion Detector
0dev.to AI
How 5 Companies Use AI Coding Agents and Assistants Across the SDLC
0dev.to DevOps
Show HN: Pip – Your Git history puts you on a Performance Improvement Plan
0Hacker News Show