Basic Examples

library(nadir)

Let’s start with an extremely simple example: a prediction problem on a continuous outcome, where we want to use cross-validation to minimize the expected risk/loss on held out data across a few different models.

We’ll use the iris dataset to do this.

nadir::super_learner() strives to keep the syntax simple, so the simplest call to super_learner() might look something like this:

super_learner(
  data = iris,
  formula = Petal.Width ~ Petal.Length + Sepal.Length + Sepal.Width,
  learners = list(lnr_lm, lnr_rf, lnr_earth, lnr_mean))
#> $predict
#> function (newdata) 
#> {
#>     Reduce(`+`, x = future_lapply(1:length(fit_learners), function(i) {
#>         fit_learners[[i]](newdata) * learner_weights[[i]]
#>     }, future.seed = TRUE))
#> }
#> <bytecode: 0x63e26ccced10>
#> <environment: 0x63e26ccd1370>
#> 
#> $y_variable
#> [1] "Petal.Width"
#> 
#> $outcome_type
#> [1] "continuous"
#> 
#> $learner_weights
#>        lm        rf     earth      mean 
#> 0.5803482 0.4196518 0.0000000 0.0000000 
#> 
#> $holdout_predictions
#> # A tibble: 150 × 6
#>    .sl_fold     lm    rf earth  mean Petal.Width
#>       <int>  <dbl> <dbl> <dbl> <dbl>       <dbl>
#>  1        1 0.201  0.268  2.25  1.19         0.2
#>  2        1 0.133  0.247  2.16  1.19         0.2
#>  3        1 0.383  0.316  2.29  1.19         0.4
#>  4        1 0.228  0.320  2.32  1.19         0.2
#>  5        1 0.316  0.332  2.28  1.19         0.5
#>  6        1 0.153  0.253  2.35  1.19         0.2
#>  7        1 0.337  0.212  1.97  1.19         0.2
#>  8        1 0.267  0.326  2.26  1.19         0.2
#>  9        1 0.0470 0.231  2.21  1.19         0.2
#> 10        1 0.0483 0.449  2.36  1.19         0.2
#> # ℹ 140 more rows
#> 
#> attr(,"class")
#> [1] "nadir_sl_model"

Notice what it returns: A function of newdata that predicts across the learners, sums up according to the learned weights, and returns the ensemble predictions.

We can store that learned predictor function and use it:

# We recommend storing more complicated arguments used repeatedly to simplify 
# the call to super_learner()
petal_formula <- Petal.Width ~ Petal.Length + Sepal.Length + Sepal.Width
learners <- list(lnr_lm, lnr_rf, lnr_earth, lnr_mean)

sl_model <- super_learner(
  data = iris,
  formula = petal_formula,
  learners = learners)

In particular, we can use it to predict on the same dataset,

predict(sl_model, iris) |> head()
#>         1         2         3         4         5         6 
#> 0.2313557 0.1800785 0.1960246 0.2454169 0.2480949 0.3774303

On a random sample of it,

predict(sl_model, iris[sample.int(size = 10, n = nrow(iris)), ]) |> 
  head()
#>       130         1       138         7        91       109 
#> 1.9375154 0.2313557 1.9929402 0.2729547 1.3948288 1.9209603

Or on completely new data.

fake_iris_data <- data.frame()
fake_iris_data <- cbind.data.frame(
  Sepal.Length = 
  rnorm(
    n = 6,
    mean = mean(iris$Sepal.Length),
    sd = sd(iris$Sepal.Length)
  ),

Sepal.Width = 
  rnorm(
    n = 6,
    mean = mean(iris$Sepal.Width),
    sd = sd(iris$Sepal.Width)
  ),

Petal.Length = 
  rnorm(
    n = 6,
    mean = mean(iris$Petal.Length),
    sd = sd(iris$Petal.Length)
  )
)

predict(sl_model, fake_iris_data) |> 
  head()
#>         1         2         3         4         5         6 
#> 1.6498660 1.3584236 0.5446859 1.8967734 2.2717115 1.2470522

Getting More Information Out

If we want to know a lot more about the super_learner() process, how it weighted the candidate learners, what the candidate learners predicted on the held-out data, etc., then we will want to look at the other metadata contained in the nadir_sl_model object produced: option.

sl_model_iris <- super_learner(
  data = iris,
  formula = petal_formula,
  learners = learners)

str(sl_model_iris, max.level = 2)
#> List of 5
#>  $ predict            :function (newdata)  
#>  $ y_variable         : chr "Petal.Width"
#>  $ outcome_type       : chr "continuous"
#>  $ learner_weights    : Named num [1:4] 5.87e-01 4.13e-01 5.96e-05 0.00
#>   ..- attr(*, "names")= chr [1:4] "lm" "rf" "earth" "mean"
#>  $ holdout_predictions: tibble [150 × 6] (S3: tbl_df/tbl/data.frame)
#>  - attr(*, "class")= chr "nadir_sl_model"

To put some description to what’s contained in the output from super_learner():

  • A prediction function, $predict() that takes newdata
  • Some character fields like $y_variable and $outcome_type to provide some context to the learning task that was performed.
  • $learner_weights that indicate what weight the different candidate learners were given
  • $holdout_predictions: A data.frame of predictions from each of the candidate learners, along with the actual outcome from the held-out data.

We can call compare_learners() on the verbose output from super_learner() if we want to assess how the different learners performed. We can also call cv_super_learner() with the same arguments as super_learner() to wrap the super_learner() call in another layer of cross-validation to assess how super_learner() performs on held-out data.

compare_learners(sl_model_iris)
#> Inferring the loss metric for learner comparison based on the outcome type:
#> outcome_type=continuous -> using mean squared error
#> # A tibble: 1 × 4
#>       lm     rf earth  mean
#>    <dbl>  <dbl> <dbl> <dbl>
#> 1 0.0384 0.0436  2.41 0.586

cv_super_learner(
  data = iris, 
  formula = petal_formula,
  learners = learners)$cv_loss
#> The loss_metric is being inferred based on the outcome_type=continuous -> using CV-MSE
#> [1] 0.03610524

We can, of course, do anything with a super learned model that we would do with a conventional prediction model, like calculating performance statistics like \(R^2\).

var_residuals <- var(iris$Sepal.Length - predict(sl_model_iris, iris))
total_variance <- var(iris$Sepal.Length)
variance_explained <- total_variance - var_residuals 

rsquared <- variance_explained / total_variance
print(rsquared)
#> [1] 0.7202571