Quantile metrics

Quantile metrics evaluate predictions that consist of predicted quantiles rather than point predictions. These metrics assess the accuracy and calibration of distributional forecasts.

Input requirements

• truth: numeric

• estimate: hardhat::quantile_pred object

Available metrics

weighted_interval_score()

Direction: minimize. Range: [0, Inf]

See also

numeric-metrics for point prediction metrics

vignette("metric-types") for an overview of all metric types

Examples

library(hardhat)

df <- data.frame(
  preds = quantile_pred(rbind(1:4, 8:11), c(0.2, 0.4, 0.6, 0.8)),
  truth = c(3.3, 7.1)
)

df
#>   preds truth
#> 1 [2.5]   3.3
#> 2 [9.5]   7.1

weighted_interval_score(df, truth, preds)
#> # A tibble: 1 × 3
#>   .metric                 .estimator .estimate
#>   <chr>                   <chr>          <dbl>
#> 1 weighted_interval_score standard        1.28