# yardstick ## Overview `yardstick` is a package to estimate how well models are working using [tidy data](https://doi.org/10.18637/jss.v059.i10) principles. See the [package webpage](https://yardstick.tidymodels.org/) for more information. ## Installation To install the package: ``` r install.packages("yardstick") # Development version: # install.packages("pak") pak::pak("tidymodels/yardstick") ``` ## Two class metric For example, suppose you create a classification model and predict on a new data set. You might have data that looks like this: ``` r library(yardstick) library(dplyr) head(two_class_example) #> truth Class1 Class2 predicted #> 1 Class2 0.00359 0.996411 Class2 #> 2 Class1 0.67862 0.321379 Class1 #> 3 Class2 0.11089 0.889106 Class2 #> 4 Class1 0.73516 0.264838 Class1 #> 5 Class2 0.01624 0.983760 Class2 #> 6 Class1 0.99928 0.000725 Class1 ``` You can use a `dplyr`-like syntax to compute common performance characteristics of the model and get them back in a data frame: ``` r metrics(two_class_example, truth, predicted) #> # A tibble: 2 × 3 #> .metric .estimator .estimate #> #> 1 accuracy binary 0.838 #> 2 kap binary 0.675 # or two_class_example |> roc_auc(truth, Class1) #> # A tibble: 1 × 3 #> .metric .estimator .estimate #> #> 1 roc_auc binary 0.939 ``` ## Multiclass metrics All classification metrics have at least one multiclass extension, with many of them having multiple ways to calculate multiclass metrics. ``` r data("hpc_cv") hpc_cv <- as_tibble(hpc_cv) hpc_cv #> # A tibble: 3,467 × 7 #> obs pred VF F M L Resample #> #> 1 VF VF 0.914 0.0779 0.00848 0.0000199 Fold01 #> 2 VF VF 0.938 0.0571 0.00482 0.0000101 Fold01 #> 3 VF VF 0.947 0.0495 0.00316 0.00000500 Fold01 #> 4 VF VF 0.929 0.0653 0.00579 0.0000156 Fold01 #> 5 VF VF 0.942 0.0543 0.00381 0.00000729 Fold01 #> 6 VF VF 0.951 0.0462 0.00272 0.00000384 Fold01 #> 7 VF VF 0.914 0.0782 0.00767 0.0000354 Fold01 #> 8 VF VF 0.918 0.0744 0.00726 0.0000157 Fold01 #> 9 VF VF 0.843 0.128 0.0296 0.000192 Fold01 #> 10 VF VF 0.920 0.0728 0.00703 0.0000147 Fold01 #> # ℹ 3,457 more rows ``` ``` r # Macro averaged multiclass precision precision(hpc_cv, obs, pred) #> # A tibble: 1 × 3 #> .metric .estimator .estimate #> #> 1 precision macro 0.631 # Micro averaged multiclass precision precision(hpc_cv, obs, pred, estimator = "micro") #> # A tibble: 1 × 3 #> .metric .estimator .estimate #> #> 1 precision micro 0.709 ``` ## Calculating metrics on resamples If you have multiple resamples of a model, you can use a metric on a grouped data frame to calculate the metric across all resamples at once. This calculates multiclass ROC AUC using the method described in Hand, Till (2001), and does it across all 10 resamples at once. ``` r hpc_cv |> group_by(Resample) |> roc_auc(obs, VF:L) #> # A tibble: 10 × 4 #> Resample .metric .estimator .estimate #> #> 1 Fold01 roc_auc hand_till 0.813 #> 2 Fold02 roc_auc hand_till 0.817 #> 3 Fold03 roc_auc hand_till 0.869 #> 4 Fold04 roc_auc hand_till 0.849 #> 5 Fold05 roc_auc hand_till 0.811 #> 6 Fold06 roc_auc hand_till 0.836 #> 7 Fold07 roc_auc hand_till 0.825 #> 8 Fold08 roc_auc hand_till 0.846 #> 9 Fold09 roc_auc hand_till 0.828 #> 10 Fold10 roc_auc hand_till 0.812 ``` ## Autoplot methods for easy visualization Curve based methods such as [`roc_curve()`](https://yardstick.tidymodels.org/reference/roc_curve.md), [`pr_curve()`](https://yardstick.tidymodels.org/reference/pr_curve.md) and [`gain_curve()`](https://yardstick.tidymodels.org/reference/gain_curve.md) all have [`ggplot2::autoplot()`](https://ggplot2.tidyverse.org/reference/autoplot.html) methods that allow for powerful and easy visualization. ``` r library(ggplot2) hpc_cv |> group_by(Resample) |> roc_curve(obs, VF:L) |> autoplot() ``` ![Faceted ROC curve. 1-specificity along the x-axis, sensitivity along the y-axis. Facets include the classes F, L, M, and VF. Each facet shows 10 lines colored to correspond to a resample. All the lines are quite overlapping. With VF having the tightest and highest values.](reference/figures/README-roc-curves-1.png) ## Contributing This project is released with a [Contributor Code of Conduct](https://contributor-covenant.org/version/2/0/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms. - For questions and discussions about tidymodels packages, modeling, and machine learning, please [post on RStudio Community](https://forum.posit.co/new-topic?category_id=15&tags=tidymodels,question). - If you think you have encountered a bug, please [submit an issue](https://github.com/tidymodels/yardstick/issues). - Either way, learn how to create and share a [reprex](https://reprex.tidyverse.org/articles/articles/learn-reprex.html) (a minimal, reproducible example), to clearly communicate about your code. - Check out further details on [contributing guidelines for tidymodels packages](https://www.tidymodels.org/contribute/) and [how to get help](https://www.tidymodels.org/help/). # Package index ## Classification Metrics - [`accuracy()`](https://yardstick.tidymodels.org/reference/accuracy.md) [`accuracy_vec()`](https://yardstick.tidymodels.org/reference/accuracy.md) : Accuracy - [`bal_accuracy()`](https://yardstick.tidymodels.org/reference/bal_accuracy.md) [`bal_accuracy_vec()`](https://yardstick.tidymodels.org/reference/bal_accuracy.md) : Balanced accuracy - [`detection_prevalence()`](https://yardstick.tidymodels.org/reference/detection_prevalence.md) [`detection_prevalence_vec()`](https://yardstick.tidymodels.org/reference/detection_prevalence.md) : Detection prevalence - [`f_meas()`](https://yardstick.tidymodels.org/reference/f_meas.md) [`f_meas_vec()`](https://yardstick.tidymodels.org/reference/f_meas.md) : F Measure - [`fall_out()`](https://yardstick.tidymodels.org/reference/fall_out.md) [`fall_out_vec()`](https://yardstick.tidymodels.org/reference/fall_out.md) : Fall-out (False Positive Rate) - [`j_index()`](https://yardstick.tidymodels.org/reference/j_index.md) [`j_index_vec()`](https://yardstick.tidymodels.org/reference/j_index.md) : J-index - [`kap()`](https://yardstick.tidymodels.org/reference/kap.md) [`kap_vec()`](https://yardstick.tidymodels.org/reference/kap.md) : Kappa - [`markedness()`](https://yardstick.tidymodels.org/reference/markedness.md) [`markedness_vec()`](https://yardstick.tidymodels.org/reference/markedness.md) : Markedness - [`mcc()`](https://yardstick.tidymodels.org/reference/mcc.md) [`mcc_vec()`](https://yardstick.tidymodels.org/reference/mcc.md) : Matthews correlation coefficient - [`miss_rate()`](https://yardstick.tidymodels.org/reference/miss_rate.md) [`miss_rate_vec()`](https://yardstick.tidymodels.org/reference/miss_rate.md) : Miss rate (False Negative Rate) - [`npv()`](https://yardstick.tidymodels.org/reference/npv.md) [`npv_vec()`](https://yardstick.tidymodels.org/reference/npv.md) : Negative predictive value - [`ppv()`](https://yardstick.tidymodels.org/reference/ppv.md) [`ppv_vec()`](https://yardstick.tidymodels.org/reference/ppv.md) : Positive predictive value - [`precision()`](https://yardstick.tidymodels.org/reference/precision.md) [`precision_vec()`](https://yardstick.tidymodels.org/reference/precision.md) : Precision - [`recall()`](https://yardstick.tidymodels.org/reference/recall.md) [`recall_vec()`](https://yardstick.tidymodels.org/reference/recall.md) : Recall - [`roc_dist()`](https://yardstick.tidymodels.org/reference/roc_dist.md) [`roc_dist_vec()`](https://yardstick.tidymodels.org/reference/roc_dist.md) : Distance to ROC corner - [`sedi()`](https://yardstick.tidymodels.org/reference/sedi.md) [`sedi_vec()`](https://yardstick.tidymodels.org/reference/sedi.md) : Symmetric Extremal Dependence Index - [`sens()`](https://yardstick.tidymodels.org/reference/sens.md) [`sens_vec()`](https://yardstick.tidymodels.org/reference/sens.md) [`sensitivity()`](https://yardstick.tidymodels.org/reference/sens.md) [`sensitivity_vec()`](https://yardstick.tidymodels.org/reference/sens.md) : Sensitivity - [`spec()`](https://yardstick.tidymodels.org/reference/spec.md) [`spec_vec()`](https://yardstick.tidymodels.org/reference/spec.md) [`specificity()`](https://yardstick.tidymodels.org/reference/spec.md) [`specificity_vec()`](https://yardstick.tidymodels.org/reference/spec.md) : Specificity ## Class Probability Metrics - [`average_precision()`](https://yardstick.tidymodels.org/reference/average_precision.md) [`average_precision_vec()`](https://yardstick.tidymodels.org/reference/average_precision.md) : Area under the precision recall curve - [`brier_class()`](https://yardstick.tidymodels.org/reference/brier_class.md) [`brier_class_vec()`](https://yardstick.tidymodels.org/reference/brier_class.md) : Brier score for classification models - [`classification_cost()`](https://yardstick.tidymodels.org/reference/classification_cost.md) [`classification_cost_vec()`](https://yardstick.tidymodels.org/reference/classification_cost.md) : Costs function for poor classification - [`gain_capture()`](https://yardstick.tidymodels.org/reference/gain_capture.md) [`gain_capture_vec()`](https://yardstick.tidymodels.org/reference/gain_capture.md) : Gain capture - [`mn_log_loss()`](https://yardstick.tidymodels.org/reference/mn_log_loss.md) [`mn_log_loss_vec()`](https://yardstick.tidymodels.org/reference/mn_log_loss.md) : Mean log loss for multinomial data - [`pr_auc()`](https://yardstick.tidymodels.org/reference/pr_auc.md) [`pr_auc_vec()`](https://yardstick.tidymodels.org/reference/pr_auc.md) : Area under the precision recall curve - [`roc_auc()`](https://yardstick.tidymodels.org/reference/roc_auc.md) [`roc_auc_vec()`](https://yardstick.tidymodels.org/reference/roc_auc.md) : Area under the receiver operator curve - [`roc_aunp()`](https://yardstick.tidymodels.org/reference/roc_aunp.md) [`roc_aunp_vec()`](https://yardstick.tidymodels.org/reference/roc_aunp.md) : Area under the ROC curve of each class against the rest, using the a priori class distribution - [`roc_aunu()`](https://yardstick.tidymodels.org/reference/roc_aunu.md) [`roc_aunu_vec()`](https://yardstick.tidymodels.org/reference/roc_aunu.md) : Area under the ROC curve of each class against the rest, using the uniform class distribution ## Ordered Probability Metrics - [`ranked_prob_score()`](https://yardstick.tidymodels.org/reference/ranked_prob_score.md) [`ranked_prob_score_vec()`](https://yardstick.tidymodels.org/reference/ranked_prob_score.md) : Ranked probability scores for ordinal classification models ## Regression Metrics - [`ccc()`](https://yardstick.tidymodels.org/reference/ccc.md) [`ccc_vec()`](https://yardstick.tidymodels.org/reference/ccc.md) : Concordance correlation coefficient - [`gini_coef()`](https://yardstick.tidymodels.org/reference/gini_coef.md) [`gini_coef_vec()`](https://yardstick.tidymodels.org/reference/gini_coef.md) : Normalized Gini coefficient - [`huber_loss()`](https://yardstick.tidymodels.org/reference/huber_loss.md) [`huber_loss_vec()`](https://yardstick.tidymodels.org/reference/huber_loss.md) : Huber loss - [`huber_loss_pseudo()`](https://yardstick.tidymodels.org/reference/huber_loss_pseudo.md) [`huber_loss_pseudo_vec()`](https://yardstick.tidymodels.org/reference/huber_loss_pseudo.md) : Psuedo-Huber Loss - [`iic()`](https://yardstick.tidymodels.org/reference/iic.md) [`iic_vec()`](https://yardstick.tidymodels.org/reference/iic.md) : Index of ideality of correlation - [`mae()`](https://yardstick.tidymodels.org/reference/mae.md) [`mae_vec()`](https://yardstick.tidymodels.org/reference/mae.md) : Mean absolute error - [`mape()`](https://yardstick.tidymodels.org/reference/mape.md) [`mape_vec()`](https://yardstick.tidymodels.org/reference/mape.md) : Mean absolute percent error - [`mase()`](https://yardstick.tidymodels.org/reference/mase.md) [`mase_vec()`](https://yardstick.tidymodels.org/reference/mase.md) : Mean absolute scaled error - [`mpe()`](https://yardstick.tidymodels.org/reference/mpe.md) [`mpe_vec()`](https://yardstick.tidymodels.org/reference/mpe.md) : Mean percentage error - [`msd()`](https://yardstick.tidymodels.org/reference/msd.md) [`msd_vec()`](https://yardstick.tidymodels.org/reference/msd.md) : Mean signed deviation - [`mse()`](https://yardstick.tidymodels.org/reference/mse.md) [`mse_vec()`](https://yardstick.tidymodels.org/reference/mse.md) : Mean squared error - [`poisson_log_loss()`](https://yardstick.tidymodels.org/reference/poisson_log_loss.md) [`poisson_log_loss_vec()`](https://yardstick.tidymodels.org/reference/poisson_log_loss.md) : Mean log loss for Poisson data - [`rmse()`](https://yardstick.tidymodels.org/reference/rmse.md) [`rmse_vec()`](https://yardstick.tidymodels.org/reference/rmse.md) : Root mean squared error - [`rmse_relative()`](https://yardstick.tidymodels.org/reference/rmse_relative.md) [`rmse_relative_vec()`](https://yardstick.tidymodels.org/reference/rmse_relative.md) : Relative root mean squared error - [`rpd()`](https://yardstick.tidymodels.org/reference/rpd.md) [`rpd_vec()`](https://yardstick.tidymodels.org/reference/rpd.md) : Ratio of performance to deviation - [`rpiq()`](https://yardstick.tidymodels.org/reference/rpiq.md) [`rpiq_vec()`](https://yardstick.tidymodels.org/reference/rpiq.md) : Ratio of performance to inter-quartile - [`rsq()`](https://yardstick.tidymodels.org/reference/rsq.md) [`rsq_vec()`](https://yardstick.tidymodels.org/reference/rsq.md) : R squared - [`rsq_trad()`](https://yardstick.tidymodels.org/reference/rsq_trad.md) [`rsq_trad_vec()`](https://yardstick.tidymodels.org/reference/rsq_trad.md) : R squared - traditional - [`smape()`](https://yardstick.tidymodels.org/reference/smape.md) [`smape_vec()`](https://yardstick.tidymodels.org/reference/smape.md) : Symmetric mean absolute percentage error ## Fairness Metrics - [`new_groupwise_metric()`](https://yardstick.tidymodels.org/reference/new_groupwise_metric.md) : Create groupwise metrics - [`demographic_parity()`](https://yardstick.tidymodels.org/reference/demographic_parity.md) : Demographic parity - [`equal_opportunity()`](https://yardstick.tidymodels.org/reference/equal_opportunity.md) : Equal opportunity - [`equalized_odds()`](https://yardstick.tidymodels.org/reference/equalized_odds.md) : Equalized odds ## Dynamic Survival Metrics - [`brier_survival()`](https://yardstick.tidymodels.org/reference/brier_survival.md) [`brier_survival_vec()`](https://yardstick.tidymodels.org/reference/brier_survival.md) : Time-Dependent Brier score for right censored data - [`brier_survival_integrated()`](https://yardstick.tidymodels.org/reference/brier_survival_integrated.md) [`brier_survival_integrated_vec()`](https://yardstick.tidymodels.org/reference/brier_survival_integrated.md) : Integrated Brier score for right censored data - [`roc_auc_survival()`](https://yardstick.tidymodels.org/reference/roc_auc_survival.md) [`roc_auc_survival_vec()`](https://yardstick.tidymodels.org/reference/roc_auc_survival.md) : Time-Dependent ROC AUC for Censored Data ## Static Survival Metrics - [`concordance_survival()`](https://yardstick.tidymodels.org/reference/concordance_survival.md) [`concordance_survival_vec()`](https://yardstick.tidymodels.org/reference/concordance_survival.md) : Concordance index for right-censored data ## Linear Predictor Survival Metrics - [`royston_survival()`](https://yardstick.tidymodels.org/reference/royston_survival.md) [`royston_survival_vec()`](https://yardstick.tidymodels.org/reference/royston_survival.md) : Royston-Sauerbei D statistic ## Curve Survival Functions - [`roc_curve_survival()`](https://yardstick.tidymodels.org/reference/roc_curve_survival.md) : Time-Dependent ROC surve for Censored Data ## Quantile Metrics - [`weighted_interval_score()`](https://yardstick.tidymodels.org/reference/weighted_interval_score.md) [`weighted_interval_score_vec()`](https://yardstick.tidymodels.org/reference/weighted_interval_score.md) : Compute weighted interval score ## Curve Functions - [`gain_curve()`](https://yardstick.tidymodels.org/reference/gain_curve.md) : Gain curve - [`lift_curve()`](https://yardstick.tidymodels.org/reference/lift_curve.md) : Lift curve - [`pr_curve()`](https://yardstick.tidymodels.org/reference/pr_curve.md) : Precision recall curve - [`roc_curve()`](https://yardstick.tidymodels.org/reference/roc_curve.md) : Receiver operator curve ## Other Functions - [`get_metrics()`](https://yardstick.tidymodels.org/reference/get_metrics.md) : Get all metrics of a given type - [`metrics()`](https://yardstick.tidymodels.org/reference/metrics.md) : General Function to Estimate Performance - [`metric_set()`](https://yardstick.tidymodels.org/reference/metric_set.md) : Combine metric functions - [`metric_tweak()`](https://yardstick.tidymodels.org/reference/metric_tweak.md) : Tweak a metric function - [`conf_mat()`](https://yardstick.tidymodels.org/reference/conf_mat.md) [`tidy(`*``*`)`](https://yardstick.tidymodels.org/reference/conf_mat.md) : Confusion Matrix for Categorical Data - [`summary(`*``*`)`](https://yardstick.tidymodels.org/reference/summary.conf_mat.md) : Summary Statistics for Confusion Matrices ## Development Functions - [`check_numeric_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) [`check_class_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) [`check_prob_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) [`check_ordered_prob_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) [`check_dynamic_survival_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) [`check_static_survival_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) [`check_linear_pred_survival_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) [`check_quantile_metric()`](https://yardstick.tidymodels.org/reference/check_metric.md) : Developer function for checking inputs in new metrics - [`dots_to_estimate()`](https://yardstick.tidymodels.org/reference/developer-helpers.md) [`get_weights()`](https://yardstick.tidymodels.org/reference/developer-helpers.md) [`finalize_estimator()`](https://yardstick.tidymodels.org/reference/developer-helpers.md) [`finalize_estimator_internal()`](https://yardstick.tidymodels.org/reference/developer-helpers.md) [`validate_estimator()`](https://yardstick.tidymodels.org/reference/developer-helpers.md) : Developer helpers - [`numeric_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`class_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`prob_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`ordered_prob_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`curve_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`dynamic_survival_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`static_survival_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`curve_survival_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`linear_pred_survival_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) [`quantile_metric_summarizer()`](https://yardstick.tidymodels.org/reference/metric-summarizers.md) : Developer function for summarizing new metrics - [`new_class_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_prob_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_ordered_prob_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_numeric_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_dynamic_survival_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_integrated_survival_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_static_survival_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_linear_pred_survival_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) [`new_quantile_metric()`](https://yardstick.tidymodels.org/reference/new-metric.md) : Construct a new metric function - [`yardstick_remove_missing()`](https://yardstick.tidymodels.org/reference/yardstick_remove_missing.md) [`yardstick_any_missing()`](https://yardstick.tidymodels.org/reference/yardstick_remove_missing.md) : Developer function for handling missing values in new metrics ## Data Sets - [`hpc_cv`](https://yardstick.tidymodels.org/reference/hpc_cv.md) : Multiclass Probability Predictions - [`lung_surv`](https://yardstick.tidymodels.org/reference/lung_surv.md) : Survival Analysis Results - [`pathology`](https://yardstick.tidymodels.org/reference/pathology.md) : Liver Pathology Data - [`solubility_test`](https://yardstick.tidymodels.org/reference/solubility_test.md) : Solubility Predictions from MARS Model - [`two_class_example`](https://yardstick.tidymodels.org/reference/two_class_example.md) : Two Class Predictions # Articles ### All vignettes - [Grouping behavior in yardstick](https://yardstick.tidymodels.org/articles/grouping.md): - [Metric types](https://yardstick.tidymodels.org/articles/metric-types.md): - [Multiclass averaging](https://yardstick.tidymodels.org/articles/multiclass.md):