Example: Model selection with optuna¶

Motivation¶

We know that model selection and/or hyperparameter optimization (HPO) can have massive impacts on the prediction quality in regular Machine Learning. Yet, it seems that model selection and hyperparameter optimization are of substantial importance for CATE estimation with MetaLearners, too, see e.g. Machlanski et. al.

However, model selection and HPO for MetaLearners look quite different from what we're used to from e.g. simple supervised learning problems. Concretely,

• In terms of a MetaLearners's option space, there are several levels to optimize for:

  1. The MetaLearner architecture, e.g. R-Learner vs DR-Learner
  2. The model to choose per base estimator of said MetaLearner architecture, e.g. LogisticRegression vs LGBMClassifier
  3. The model hyperparameters per base model

• On a conceptual level, it's not clear how to measure model quality for MetaLearners. As a proxy for the underlying quantity of interest one might look into base model performance, the R-Loss of the CATE estimates or some more elaborate approaches alluded to by Machlanski et. al.

We think that HPO can be divided into two camps:

• Exploration of (hyperparameter, metric evaluation) pairs where the pairs do not influence each other (e.g. grid search, random search)

• Exploration of (hyperparameter, metric evaluation) pairs where the pairs do influence each other (e.g. Bayesian optimization, evolutionary algorithms); in other words, there is a feedback-loop between sample result and sample

In this example, we will illustrate the latter camp based on an application of optuna -- a popular framework for HPO -- in interplay with metalearners. For the former please refer to the example on hyperparameter tuning with MetaLearnerGridSearch.

Installation¶

In order to use optuna, we first need to install the package. We can do so either via conda and conda-forge

$ conda install optuna -c conda-forge

or via pip and PyPI

$ pip install optuna

Usage¶

Loading the data¶

Just like in our example on estimating CATEs with a MetaLearner, we will first load some experiment data:

In [1]:

import pandas as pd
from pathlib import Path
from git_root import git_root

df = pd.read_csv(git_root("data/learning_mindset.zip"))
outcome_column = "achievement_score"
treatment_column = "intervention"
feature_columns = [
    column for column in df.columns if column not in [outcome_column, treatment_column]
]
categorical_feature_columns = [
    "ethnicity",
    "gender",
    "frst_in_family",
    "school_urbanicity",
    "schoolid",
]
# Note that explicitly setting the dtype of these features to category
# allows both lightgbm as well as shap plots to
# 1. Operate on features which are not of type int, bool or float
# 2. Correctly interpret categoricals with int values to be
#    interpreted as categoricals, as compared to ordinals/numericals.
for categorical_feature_column in categorical_feature_columns:
    df[categorical_feature_column] = df[categorical_feature_column].astype("category")

import pandas as pd from pathlib import Path from git_root import git_root df = pd.read_csv(git_root("data/learning_mindset.zip")) outcome_column = "achievement_score" treatment_column = "intervention" feature_columns = [ column for column in df.columns if column not in [outcome_column, treatment_column] ] categorical_feature_columns = [ "ethnicity", "gender", "frst_in_family", "school_urbanicity", "schoolid", ] # Note that explicitly setting the dtype of these features to category # allows both lightgbm as well as shap plots to # 1. Operate on features which are not of type int, bool or float # 2. Correctly interpret categoricals with int values to be # interpreted as categoricals, as compared to ordinals/numericals. for categorical_feature_column in categorical_feature_columns: df[categorical_feature_column] = df[categorical_feature_column].astype("category")

Now that we've loaded the experiment data, we can split it up into train and validation data:

In [2]:

from sklearn.model_selection import train_test_split

X_train, X_validation, y_train, y_validation, w_train, w_validation = train_test_split(
    df[feature_columns], df[outcome_column], df[treatment_column], test_size=0.25
)

from sklearn.model_selection import train_test_split X_train, X_validation, y_train, y_validation, w_train, w_validation = train_test_split( df[feature_columns], df[outcome_column], df[treatment_column], test_size=0.25 )

Optimizing base model hyperparameters¶

Let's say that we want to work with an RLearner and LightGBM estimators for base models. We will seek two optimize three hyperparameters of our base models:

• The number of estimators n_estimators of our outcome model.
• The max depth max_depth of our outcome model.
• The number of estimators n_estimators of our treatment effect model.

We can mold this ambition into the following simple script creating an optuna study:

In [3]:

import optuna
from metalearners.rlearner import r_loss
from metalearners.utils import simplify_output
from metalearners import RLearner
from lightgbm import LGBMRegressor, LGBMClassifier


def objective(trial):

    n_estimators_nuisance = trial.suggest_int("n_estimators_nuisance", 5, 250)
    max_depth_nuisance = trial.suggest_int("max_depth_nuisance", 3, 30)
    n_estimators_treatment = trial.suggest_int("n_estimators_treatment", 5, 100)

    rlearner = RLearner(
        nuisance_model_factory=LGBMRegressor,
        nuisance_model_params={
            "n_estimators": n_estimators_nuisance,
            "max_depth": max_depth_nuisance,
            "verbosity": -1,
        },
        propensity_model_factory=LGBMClassifier,
        propensity_model_params={"n_estimators": 5, "verbosity": -1},
        treatment_model_factory=LGBMRegressor,
        treatment_model_params={
            "n_estimators": n_estimators_treatment,
            "verbosity": -1,
        },
        is_classification=False,
        n_variants=2,
    )

    rlearner.fit(X=X_train, y=y_train, w=w_train)

    return rlearner.evaluate(
        X=X_validation,
        y=y_validation,
        w=w_validation,
        is_oos=True,
    )["r_loss_1_vs_0"]


study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=100)

import optuna from metalearners.rlearner import r_loss from metalearners.utils import simplify_output from metalearners import RLearner from lightgbm import LGBMRegressor, LGBMClassifier def objective(trial): n_estimators_nuisance = trial.suggest_int("n_estimators_nuisance", 5, 250) max_depth_nuisance = trial.suggest_int("max_depth_nuisance", 3, 30) n_estimators_treatment = trial.suggest_int("n_estimators_treatment", 5, 100) rlearner = RLearner( nuisance_model_factory=LGBMRegressor, nuisance_model_params={ "n_estimators": n_estimators_nuisance, "max_depth": max_depth_nuisance, "verbosity": -1, }, propensity_model_factory=LGBMClassifier, propensity_model_params={"n_estimators": 5, "verbosity": -1}, treatment_model_factory=LGBMRegressor, treatment_model_params={ "n_estimators": n_estimators_treatment, "verbosity": -1, }, is_classification=False, n_variants=2, ) rlearner.fit(X=X_train, y=y_train, w=w_train) return rlearner.evaluate( X=X_validation, y=y_validation, w=w_validation, is_oos=True, )["r_loss_1_vs_0"] study = optuna.create_study(direction="minimize") study.optimize(objective, n_trials=100)

[I 2024-06-24 13:22:16,992] A new study created in memory with name: no-name-91d5c937-f1af-49ed-80c9-85ae6b2fef65
[I 2024-06-24 13:22:22,633] Trial 0 finished with value: 0.83053857095663 and parameters: {'n_estimators_nuisance': 142, 'max_depth_nuisance': 29, 'n_estimators_treatment': 8}. Best is trial 0 with value: 0.83053857095663.
[I 2024-06-24 13:22:27,327] Trial 1 finished with value: 0.8279630501887009 and parameters: {'n_estimators_nuisance': 89, 'max_depth_nuisance': 19, 'n_estimators_treatment': 39}. Best is trial 1 with value: 0.8279630501887009.
[I 2024-06-24 13:22:36,426] Trial 2 finished with value: 0.8438524520544455 and parameters: {'n_estimators_nuisance': 223, 'max_depth_nuisance': 28, 'n_estimators_treatment': 48}. Best is trial 1 with value: 0.8279630501887009.
[I 2024-06-24 13:22:42,400] Trial 3 finished with value: 0.8336463393551226 and parameters: {'n_estimators_nuisance': 87, 'max_depth_nuisance': 22, 'n_estimators_treatment': 76}. Best is trial 1 with value: 0.8279630501887009.
[I 2024-06-24 13:22:51,339] Trial 4 finished with value: 0.8458746918691048 and parameters: {'n_estimators_nuisance': 190, 'max_depth_nuisance': 24, 'n_estimators_treatment': 71}. Best is trial 1 with value: 0.8279630501887009.
[I 2024-06-24 13:22:55,459] Trial 5 finished with value: 0.8255242541019684 and parameters: {'n_estimators_nuisance': 86, 'max_depth_nuisance': 14, 'n_estimators_treatment': 18}. Best is trial 5 with value: 0.8255242541019684.
[I 2024-06-24 13:23:01,144] Trial 6 finished with value: 0.8300992723506675 and parameters: {'n_estimators_nuisance': 54, 'max_depth_nuisance': 22, 'n_estimators_treatment': 99}. Best is trial 5 with value: 0.8255242541019684.
[I 2024-06-24 13:23:08,313] Trial 7 finished with value: 0.8359314753840047 and parameters: {'n_estimators_nuisance': 157, 'max_depth_nuisance': 30, 'n_estimators_treatment': 44}. Best is trial 5 with value: 0.8255242541019684.
[I 2024-06-24 13:23:18,652] Trial 8 finished with value: 0.8526731227345625 and parameters: {'n_estimators_nuisance': 207, 'max_depth_nuisance': 30, 'n_estimators_treatment': 100}. Best is trial 5 with value: 0.8255242541019684.
[I 2024-06-24 13:23:27,079] Trial 9 finished with value: 0.8415862061713413 and parameters: {'n_estimators_nuisance': 177, 'max_depth_nuisance': 27, 'n_estimators_treatment': 65}. Best is trial 5 with value: 0.8255242541019684.
[I 2024-06-24 13:23:31,460] Trial 10 finished with value: 0.8250971170471355 and parameters: {'n_estimators_nuisance': 105, 'max_depth_nuisance': 9, 'n_estimators_treatment': 10}. Best is trial 10 with value: 0.8250971170471355.
[I 2024-06-24 13:23:32,436] Trial 11 finished with value: 0.8574556785022519 and parameters: {'n_estimators_nuisance': 7, 'max_depth_nuisance': 9, 'n_estimators_treatment': 7}. Best is trial 10 with value: 0.8250971170471355.
[I 2024-06-24 13:23:37,210] Trial 12 finished with value: 0.8288622966745569 and parameters: {'n_estimators_nuisance': 101, 'max_depth_nuisance': 11, 'n_estimators_treatment': 24}. Best is trial 10 with value: 0.8250971170471355.
[I 2024-06-24 13:23:39,330] Trial 13 finished with value: 0.815499188054067 and parameters: {'n_estimators_nuisance': 47, 'max_depth_nuisance': 4, 'n_estimators_treatment': 25}. Best is trial 13 with value: 0.815499188054067.
[I 2024-06-24 13:23:41,045] Trial 14 finished with value: 0.8170278638514555 and parameters: {'n_estimators_nuisance': 32, 'max_depth_nuisance': 3, 'n_estimators_treatment': 27}. Best is trial 13 with value: 0.815499188054067.
[I 2024-06-24 13:23:42,685] Trial 15 finished with value: 0.8840084413491256 and parameters: {'n_estimators_nuisance': 6, 'max_depth_nuisance': 3, 'n_estimators_treatment': 31}. Best is trial 13 with value: 0.815499188054067.
[I 2024-06-24 13:23:44,543] Trial 16 finished with value: 0.8133591571184091 and parameters: {'n_estimators_nuisance': 45, 'max_depth_nuisance': 3, 'n_estimators_treatment': 28}. Best is trial 16 with value: 0.8133591571184091.
[I 2024-06-24 13:23:48,523] Trial 17 finished with value: 0.8225422607328783 and parameters: {'n_estimators_nuisance': 43, 'max_depth_nuisance': 6, 'n_estimators_treatment': 58}. Best is trial 16 with value: 0.8133591571184091.
[I 2024-06-24 13:23:52,255] Trial 18 finished with value: 0.8232256196643117 and parameters: {'n_estimators_nuisance': 62, 'max_depth_nuisance': 6, 'n_estimators_treatment': 36}. Best is trial 16 with value: 0.8133591571184091.
[I 2024-06-24 13:24:01,178] Trial 19 finished with value: 0.8424506982382707 and parameters: {'n_estimators_nuisance': 245, 'max_depth_nuisance': 14, 'n_estimators_treatment': 19}. Best is trial 16 with value: 0.8133591571184091.
[I 2024-06-24 13:24:04,392] Trial 20 finished with value: 0.8214218530466108 and parameters: {'n_estimators_nuisance': 28, 'max_depth_nuisance': 6, 'n_estimators_treatment': 51}. Best is trial 16 with value: 0.8133591571184091.
[I 2024-06-24 13:24:06,091] Trial 21 finished with value: 0.8159680095390245 and parameters: {'n_estimators_nuisance': 32, 'max_depth_nuisance': 3, 'n_estimators_treatment': 25}. Best is trial 16 with value: 0.8133591571184091.
[I 2024-06-24 13:24:07,765] Trial 22 finished with value: 0.8128120242383954 and parameters: {'n_estimators_nuisance': 66, 'max_depth_nuisance': 3, 'n_estimators_treatment': 17}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:11,106] Trial 23 finished with value: 0.8222947299276746 and parameters: {'n_estimators_nuisance': 67, 'max_depth_nuisance': 8, 'n_estimators_treatment': 16}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:16,844] Trial 24 finished with value: 0.8319423296916826 and parameters: {'n_estimators_nuisance': 120, 'max_depth_nuisance': 12, 'n_estimators_treatment': 35}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:19,600] Trial 25 finished with value: 0.8189416518668202 and parameters: {'n_estimators_nuisance': 69, 'max_depth_nuisance': 5, 'n_estimators_treatment': 14}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:22,457] Trial 26 finished with value: 0.8217825547992809 and parameters: {'n_estimators_nuisance': 22, 'max_depth_nuisance': 8, 'n_estimators_treatment': 43}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:25,403] Trial 27 finished with value: 0.8177032033179947 and parameters: {'n_estimators_nuisance': 49, 'max_depth_nuisance': 5, 'n_estimators_treatment': 28}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:30,223] Trial 28 finished with value: 0.8276434740695241 and parameters: {'n_estimators_nuisance': 117, 'max_depth_nuisance': 12, 'n_estimators_treatment': 5}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:36,368] Trial 29 finished with value: 0.8328930801551955 and parameters: {'n_estimators_nuisance': 149, 'max_depth_nuisance': 16, 'n_estimators_treatment': 22}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:40,779] Trial 30 finished with value: 0.8249268400596328 and parameters: {'n_estimators_nuisance': 134, 'max_depth_nuisance': 4, 'n_estimators_treatment': 58}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:42,679] Trial 31 finished with value: 0.8151490160047353 and parameters: {'n_estimators_nuisance': 35, 'max_depth_nuisance': 3, 'n_estimators_treatment': 32}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:46,846] Trial 32 finished with value: 0.8277404652894443 and parameters: {'n_estimators_nuisance': 76, 'max_depth_nuisance': 7, 'n_estimators_treatment': 33}. Best is trial 22 with value: 0.8128120242383954.
[I 2024-06-24 13:24:48,143] Trial 33 finished with value: 0.8128115911320891 and parameters: {'n_estimators_nuisance': 42, 'max_depth_nuisance': 3, 'n_estimators_treatment': 12}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:24:49,186] Trial 34 finished with value: 0.8248887136761331 and parameters: {'n_estimators_nuisance': 19, 'max_depth_nuisance': 3, 'n_estimators_treatment': 11}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:24:52,592] Trial 35 finished with value: 0.8197588967136334 and parameters: {'n_estimators_nuisance': 41, 'max_depth_nuisance': 10, 'n_estimators_treatment': 40}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:24:56,462] Trial 36 finished with value: 0.8234905128396314 and parameters: {'n_estimators_nuisance': 86, 'max_depth_nuisance': 19, 'n_estimators_treatment': 13}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:24:59,230] Trial 37 finished with value: 0.8195390479856022 and parameters: {'n_estimators_nuisance': 57, 'max_depth_nuisance': 5, 'n_estimators_treatment': 20}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:25:03,395] Trial 38 finished with value: 0.8279481584510203 and parameters: {'n_estimators_nuisance': 77, 'max_depth_nuisance': 7, 'n_estimators_treatment': 30}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:25:08,099] Trial 39 finished with value: 0.8289172796994028 and parameters: {'n_estimators_nuisance': 100, 'max_depth_nuisance': 4, 'n_estimators_treatment': 86}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:25:10,867] Trial 40 finished with value: 0.826447064285644 and parameters: {'n_estimators_nuisance': 16, 'max_depth_nuisance': 7, 'n_estimators_treatment': 47}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:25:12,632] Trial 41 finished with value: 0.8134732587357059 and parameters: {'n_estimators_nuisance': 40, 'max_depth_nuisance': 4, 'n_estimators_treatment': 17}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:25:14,751] Trial 42 finished with value: 0.8153749119409206 and parameters: {'n_estimators_nuisance': 40, 'max_depth_nuisance': 5, 'n_estimators_treatment': 16}. Best is trial 33 with value: 0.8128115911320891.
[I 2024-06-24 13:25:15,981] Trial 43 finished with value: 0.8119776836172341 and parameters: {'n_estimators_nuisance': 55, 'max_depth_nuisance': 3, 'n_estimators_treatment': 8}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:18,577] Trial 44 finished with value: 0.8197695823883865 and parameters: {'n_estimators_nuisance': 56, 'max_depth_nuisance': 26, 'n_estimators_treatment': 5}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:20,525] Trial 45 finished with value: 0.8148575205546763 and parameters: {'n_estimators_nuisance': 76, 'max_depth_nuisance': 4, 'n_estimators_treatment': 9}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:24,558] Trial 46 finished with value: 0.8235706812289811 and parameters: {'n_estimators_nuisance': 88, 'max_depth_nuisance': 21, 'n_estimators_treatment': 11}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:26,326] Trial 47 finished with value: 0.8247959972665566 and parameters: {'n_estimators_nuisance': 15, 'max_depth_nuisance': 9, 'n_estimators_treatment': 20}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:28,979] Trial 48 finished with value: 0.8199357058869834 and parameters: {'n_estimators_nuisance': 51, 'max_depth_nuisance': 6, 'n_estimators_treatment': 16}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:30,308] Trial 49 finished with value: 0.8125109202263244 and parameters: {'n_estimators_nuisance': 64, 'max_depth_nuisance': 3, 'n_estimators_treatment': 8}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:34,563] Trial 50 finished with value: 0.826890686161164 and parameters: {'n_estimators_nuisance': 100, 'max_depth_nuisance': 8, 'n_estimators_treatment': 9}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:36,069] Trial 51 finished with value: 0.813546831729049 and parameters: {'n_estimators_nuisance': 65, 'max_depth_nuisance': 3, 'n_estimators_treatment': 13}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:38,167] Trial 52 finished with value: 0.817531733623378 and parameters: {'n_estimators_nuisance': 54, 'max_depth_nuisance': 5, 'n_estimators_treatment': 5}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:41,914] Trial 53 finished with value: 0.8244505669256544 and parameters: {'n_estimators_nuisance': 181, 'max_depth_nuisance': 4, 'n_estimators_treatment': 23}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:43,376] Trial 54 finished with value: 0.81444877687994 and parameters: {'n_estimators_nuisance': 30, 'max_depth_nuisance': 3, 'n_estimators_treatment': 19}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:45,502] Trial 55 finished with value: 0.8162547649658873 and parameters: {'n_estimators_nuisance': 44, 'max_depth_nuisance': 6, 'n_estimators_treatment': 8}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:47,781] Trial 56 finished with value: 0.8149805650123045 and parameters: {'n_estimators_nuisance': 61, 'max_depth_nuisance': 4, 'n_estimators_treatment': 15}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:53,264] Trial 57 finished with value: 0.8315096749793097 and parameters: {'n_estimators_nuisance': 73, 'max_depth_nuisance': 7, 'n_estimators_treatment': 76}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:55,067] Trial 58 finished with value: 0.8385232810292145 and parameters: {'n_estimators_nuisance': 10, 'max_depth_nuisance': 5, 'n_estimators_treatment': 28}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:25:56,696] Trial 59 finished with value: 0.8198848701778038 and parameters: {'n_estimators_nuisance': 24, 'max_depth_nuisance': 3, 'n_estimators_treatment': 25}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:00,539] Trial 60 finished with value: 0.8229817488670822 and parameters: {'n_estimators_nuisance': 84, 'max_depth_nuisance': 10, 'n_estimators_treatment': 12}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:02,186] Trial 61 finished with value: 0.8132559375635982 and parameters: {'n_estimators_nuisance': 63, 'max_depth_nuisance': 3, 'n_estimators_treatment': 17}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:03,906] Trial 62 finished with value: 0.8135751077653381 and parameters: {'n_estimators_nuisance': 38, 'max_depth_nuisance': 4, 'n_estimators_treatment': 18}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:06,739] Trial 63 finished with value: 0.820160456617976 and parameters: {'n_estimators_nuisance': 65, 'max_depth_nuisance': 6, 'n_estimators_treatment': 8}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:08,423] Trial 64 finished with value: 0.8137274457723414 and parameters: {'n_estimators_nuisance': 49, 'max_depth_nuisance': 3, 'n_estimators_treatment': 22}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:11,796] Trial 65 finished with value: 0.8219241473121046 and parameters: {'n_estimators_nuisance': 94, 'max_depth_nuisance': 5, 'n_estimators_treatment': 17}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:15,538] Trial 66 finished with value: 0.8241552798958554 and parameters: {'n_estimators_nuisance': 212, 'max_depth_nuisance': 4, 'n_estimators_treatment': 12}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:17,022] Trial 67 finished with value: 0.8179352543703808 and parameters: {'n_estimators_nuisance': 27, 'max_depth_nuisance': 3, 'n_estimators_treatment': 22}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:19,674] Trial 68 finished with value: 0.8214883150956522 and parameters: {'n_estimators_nuisance': 58, 'max_depth_nuisance': 7, 'n_estimators_treatment': 7}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:24,398] Trial 69 finished with value: 0.8276503443147979 and parameters: {'n_estimators_nuisance': 109, 'max_depth_nuisance': 15, 'n_estimators_treatment': 14}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:27,859] Trial 70 finished with value: 0.8222121483908412 and parameters: {'n_estimators_nuisance': 45, 'max_depth_nuisance': 19, 'n_estimators_treatment': 37}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:29,355] Trial 71 finished with value: 0.8137150744907906 and parameters: {'n_estimators_nuisance': 68, 'max_depth_nuisance': 3, 'n_estimators_treatment': 12}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:31,538] Trial 72 finished with value: 0.8161224230955217 and parameters: {'n_estimators_nuisance': 81, 'max_depth_nuisance': 4, 'n_estimators_treatment': 14}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:33,967] Trial 73 finished with value: 0.8163451474418802 and parameters: {'n_estimators_nuisance': 35, 'max_depth_nuisance': 6, 'n_estimators_treatment': 19}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:37,042] Trial 74 finished with value: 0.8207879229295765 and parameters: {'n_estimators_nuisance': 67, 'max_depth_nuisance': 5, 'n_estimators_treatment': 25}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:42,050] Trial 75 finished with value: 0.8249375102788891 and parameters: {'n_estimators_nuisance': 161, 'max_depth_nuisance': 3, 'n_estimators_treatment': 91}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:43,866] Trial 76 finished with value: 0.814351699028666 and parameters: {'n_estimators_nuisance': 61, 'max_depth_nuisance': 4, 'n_estimators_treatment': 10}. Best is trial 43 with value: 0.8119776836172341.
[I 2024-06-24 13:26:44,910] Trial 77 finished with value: 0.8116086796644417 and parameters: {'n_estimators_nuisance': 48, 'max_depth_nuisance': 3, 'n_estimators_treatment': 5}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:26:46,951] Trial 78 finished with value: 0.8162769369923417 and parameters: {'n_estimators_nuisance': 50, 'max_depth_nuisance': 5, 'n_estimators_treatment': 5}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:26:49,055] Trial 79 finished with value: 0.8165225847539919 and parameters: {'n_estimators_nuisance': 35, 'max_depth_nuisance': 8, 'n_estimators_treatment': 7}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:26:53,550] Trial 80 finished with value: 0.8256028532248583 and parameters: {'n_estimators_nuisance': 94, 'max_depth_nuisance': 24, 'n_estimators_treatment': 16}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:26:54,757] Trial 81 finished with value: 0.8128688330088947 and parameters: {'n_estimators_nuisance': 42, 'max_depth_nuisance': 3, 'n_estimators_treatment': 10}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:26:56,003] Trial 82 finished with value: 0.8124509466640654 and parameters: {'n_estimators_nuisance': 43, 'max_depth_nuisance': 3, 'n_estimators_treatment': 10}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:26:57,320] Trial 83 finished with value: 0.8121261895285019 and parameters: {'n_estimators_nuisance': 55, 'max_depth_nuisance': 3, 'n_estimators_treatment': 9}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:26:58,805] Trial 84 finished with value: 0.812766344834344 and parameters: {'n_estimators_nuisance': 72, 'max_depth_nuisance': 3, 'n_estimators_treatment': 10}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:00,499] Trial 85 finished with value: 0.8136283539767701 and parameters: {'n_estimators_nuisance': 54, 'max_depth_nuisance': 4, 'n_estimators_treatment': 10}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:03,034] Trial 86 finished with value: 0.8187490935709931 and parameters: {'n_estimators_nuisance': 72, 'max_depth_nuisance': 5, 'n_estimators_treatment': 7}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:05,379] Trial 87 finished with value: 0.8185500681713066 and parameters: {'n_estimators_nuisance': 46, 'max_depth_nuisance': 6, 'n_estimators_treatment': 9}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:06,709] Trial 88 finished with value: 0.8128958351668335 and parameters: {'n_estimators_nuisance': 80, 'max_depth_nuisance': 3, 'n_estimators_treatment': 5}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:08,115] Trial 89 finished with value: 0.8125643450061724 and parameters: {'n_estimators_nuisance': 32, 'max_depth_nuisance': 4, 'n_estimators_treatment': 11}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:11,113] Trial 90 finished with value: 0.819571266992025 and parameters: {'n_estimators_nuisance': 30, 'max_depth_nuisance': 4, 'n_estimators_treatment': 60}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:12,190] Trial 91 finished with value: 0.8211563812138972 and parameters: {'n_estimators_nuisance': 23, 'max_depth_nuisance': 3, 'n_estimators_treatment': 11}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:14,122] Trial 92 finished with value: 0.814698892556723 and parameters: {'n_estimators_nuisance': 42, 'max_depth_nuisance': 5, 'n_estimators_treatment': 7}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:15,961] Trial 93 finished with value: 0.8141108073354297 and parameters: {'n_estimators_nuisance': 54, 'max_depth_nuisance': 4, 'n_estimators_treatment': 14}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:16,887] Trial 94 finished with value: 0.8381580722873854 and parameters: {'n_estimators_nuisance': 13, 'max_depth_nuisance': 3, 'n_estimators_treatment': 10}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:24,514] Trial 95 finished with value: 0.8408935470945624 and parameters: {'n_estimators_nuisance': 247, 'max_depth_nuisance': 6, 'n_estimators_treatment': 7}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:26,544] Trial 96 finished with value: 0.8134953881569535 and parameters: {'n_estimators_nuisance': 36, 'max_depth_nuisance': 5, 'n_estimators_treatment': 13}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:27,691] Trial 97 finished with value: 0.812240831092427 and parameters: {'n_estimators_nuisance': 58, 'max_depth_nuisance': 3, 'n_estimators_treatment': 5}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:29,225] Trial 98 finished with value: 0.8147806106268247 and parameters: {'n_estimators_nuisance': 58, 'max_depth_nuisance': 4, 'n_estimators_treatment': 5}. Best is trial 77 with value: 0.8116086796644417.
[I 2024-06-24 13:27:32,568] Trial 99 finished with value: 0.82155698256571 and parameters: {'n_estimators_nuisance': 72, 'max_depth_nuisance': 13, 'n_estimators_treatment': 8}. Best is trial 77 with value: 0.8116086796644417.

Note that the metric to be optimized is the R-Loss here. We can obtain it -- among other metrics -- via the RLearner's evaluate method. We can see it evolve as follows across the 100 trials.

Alternatively, if we'd like to optimize a base model in light of its individual metric -- in this case an RMSE on the observed outcomes for an the outcome model -- we can easily do that, too:

In [4]:

from sklearn.metrics import root_mean_squared_error


def objective_individual(trial):

    n_estimators_nuisance = trial.suggest_int("n_estimators_nuisance", 5, 250)
    max_depth_nuisance = trial.suggest_int("max_depth_nuisance", 3, 30)

    rlearner = RLearner(
        nuisance_model_factory=LGBMRegressor,
        nuisance_model_params={
            "n_estimators": n_estimators_nuisance,
            "max_depth": max_depth_nuisance,
            "verbosity": -1,
        },
        propensity_model_factory=LGBMClassifier,
        treatment_model_factory=LGBMRegressor,
        is_classification=False,
        n_variants=2,
    )

    rlearner.fit_nuisance(X=X_train, y=y_train, model_kind="outcome_model", model_ord=0)

    outcome_predictions = rlearner.predict_nuisance(
        X=X_validation, model_kind="outcome_model", model_ord=0, is_oos=True
    )

    return root_mean_squared_error(y_validation, outcome_predictions)


study_individual = optuna.create_study(direction="minimize")
study_individual.optimize(objective_individual, n_trials=100)

from sklearn.metrics import root_mean_squared_error def objective_individual(trial): n_estimators_nuisance = trial.suggest_int("n_estimators_nuisance", 5, 250) max_depth_nuisance = trial.suggest_int("max_depth_nuisance", 3, 30) rlearner = RLearner( nuisance_model_factory=LGBMRegressor, nuisance_model_params={ "n_estimators": n_estimators_nuisance, "max_depth": max_depth_nuisance, "verbosity": -1, }, propensity_model_factory=LGBMClassifier, treatment_model_factory=LGBMRegressor, is_classification=False, n_variants=2, ) rlearner.fit_nuisance(X=X_train, y=y_train, model_kind="outcome_model", model_ord=0) outcome_predictions = rlearner.predict_nuisance( X=X_validation, model_kind="outcome_model", model_ord=0, is_oos=True ) return root_mean_squared_error(y_validation, outcome_predictions) study_individual = optuna.create_study(direction="minimize") study_individual.optimize(objective_individual, n_trials=100)

[I 2024-06-24 13:27:32,573] A new study created in memory with name: no-name-c8e6fb60-ecbd-4bf5-95f8-fdbea711e7d2
[I 2024-06-24 13:27:37,408] Trial 0 finished with value: 0.8557745599990257 and parameters: {'n_estimators_nuisance': 140, 'max_depth_nuisance': 14}. Best is trial 0 with value: 0.8557745599990257.
[I 2024-06-24 13:27:44,610] Trial 1 finished with value: 0.8645415696834704 and parameters: {'n_estimators_nuisance': 225, 'max_depth_nuisance': 10}. Best is trial 0 with value: 0.8557745599990257.
[I 2024-06-24 13:27:51,077] Trial 2 finished with value: 0.8608539811229967 and parameters: {'n_estimators_nuisance': 197, 'max_depth_nuisance': 14}. Best is trial 0 with value: 0.8557745599990257.
[I 2024-06-24 13:27:53,395] Trial 3 finished with value: 0.8463791376788804 and parameters: {'n_estimators_nuisance': 73, 'max_depth_nuisance': 6}. Best is trial 3 with value: 0.8463791376788804.
[I 2024-06-24 13:27:57,908] Trial 4 finished with value: 0.8555935808124866 and parameters: {'n_estimators_nuisance': 136, 'max_depth_nuisance': 24}. Best is trial 3 with value: 0.8463791376788804.
[I 2024-06-24 13:28:02,562] Trial 5 finished with value: 0.8575332566164642 and parameters: {'n_estimators_nuisance': 166, 'max_depth_nuisance': 6}. Best is trial 3 with value: 0.8463791376788804.
[I 2024-06-24 13:28:06,781] Trial 6 finished with value: 0.8540343236087775 and parameters: {'n_estimators_nuisance': 124, 'max_depth_nuisance': 26}. Best is trial 3 with value: 0.8463791376788804.
[I 2024-06-24 13:28:11,356] Trial 7 finished with value: 0.8556747079702898 and parameters: {'n_estimators_nuisance': 138, 'max_depth_nuisance': 30}. Best is trial 3 with value: 0.8463791376788804.
[I 2024-06-24 13:28:11,905] Trial 8 finished with value: 0.8542846197500849 and parameters: {'n_estimators_nuisance': 12, 'max_depth_nuisance': 9}. Best is trial 3 with value: 0.8463791376788804.
[I 2024-06-24 13:28:16,303] Trial 9 finished with value: 0.8546913754562475 and parameters: {'n_estimators_nuisance': 130, 'max_depth_nuisance': 14}. Best is trial 3 with value: 0.8463791376788804.
[I 2024-06-24 13:28:16,844] Trial 10 finished with value: 0.8371027723046572 and parameters: {'n_estimators_nuisance': 54, 'max_depth_nuisance': 3}. Best is trial 10 with value: 0.8371027723046572.
[I 2024-06-24 13:28:17,339] Trial 11 finished with value: 0.8369717349425706 and parameters: {'n_estimators_nuisance': 48, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:18,831] Trial 12 finished with value: 0.8421319598691644 and parameters: {'n_estimators_nuisance': 43, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:19,587] Trial 13 finished with value: 0.8380130474396555 and parameters: {'n_estimators_nuisance': 81, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:19,735] Trial 14 finished with value: 0.8788175155940677 and parameters: {'n_estimators_nuisance': 9, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:22,341] Trial 15 finished with value: 0.8478549784624454 and parameters: {'n_estimators_nuisance': 73, 'max_depth_nuisance': 19}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:23,965] Trial 16 finished with value: 0.8423566325073685 and parameters: {'n_estimators_nuisance': 41, 'max_depth_nuisance': 9}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:27,109] Trial 17 finished with value: 0.8497453274590234 and parameters: {'n_estimators_nuisance': 87, 'max_depth_nuisance': 18}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:28,627] Trial 18 finished with value: 0.841769145993798 and parameters: {'n_estimators_nuisance': 39, 'max_depth_nuisance': 11}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:29,529] Trial 19 finished with value: 0.8387476993794126 and parameters: {'n_estimators_nuisance': 100, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:31,383] Trial 20 finished with value: 0.8453709840781217 and parameters: {'n_estimators_nuisance': 54, 'max_depth_nuisance': 7}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:32,278] Trial 21 finished with value: 0.8387476993794126 and parameters: {'n_estimators_nuisance': 100, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:32,944] Trial 22 finished with value: 0.8371994900203269 and parameters: {'n_estimators_nuisance': 64, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:33,949] Trial 23 finished with value: 0.8403547831901805 and parameters: {'n_estimators_nuisance': 26, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:36,213] Trial 24 finished with value: 0.8468467370952899 and parameters: {'n_estimators_nuisance': 63, 'max_depth_nuisance': 8}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:39,863] Trial 25 finished with value: 0.851761674155948 and parameters: {'n_estimators_nuisance': 103, 'max_depth_nuisance': 12}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:40,798] Trial 26 finished with value: 0.8392291117844327 and parameters: {'n_estimators_nuisance': 27, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:42,850] Trial 27 finished with value: 0.8449490009990133 and parameters: {'n_estimators_nuisance': 56, 'max_depth_nuisance': 21}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:44,426] Trial 28 finished with value: 0.8415815451435248 and parameters: {'n_estimators_nuisance': 112, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:49,544] Trial 29 finished with value: 0.8565679958272611 and parameters: {'n_estimators_nuisance': 153, 'max_depth_nuisance': 13}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:50,579] Trial 30 finished with value: 0.8410849898275524 and parameters: {'n_estimators_nuisance': 24, 'max_depth_nuisance': 16}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:51,794] Trial 31 finished with value: 0.8407693671099681 and parameters: {'n_estimators_nuisance': 78, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:54,786] Trial 32 finished with value: 0.8508008145559798 and parameters: {'n_estimators_nuisance': 86, 'max_depth_nuisance': 8}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:56,515] Trial 33 finished with value: 0.842614606566614 and parameters: {'n_estimators_nuisance': 59, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:28:58,200] Trial 34 finished with value: 0.8425035720305093 and parameters: {'n_estimators_nuisance': 208, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:00,761] Trial 35 finished with value: 0.8465773772106769 and parameters: {'n_estimators_nuisance': 69, 'max_depth_nuisance': 10}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:06,045] Trial 36 finished with value: 0.8612283843050235 and parameters: {'n_estimators_nuisance': 240, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:07,732] Trial 37 finished with value: 0.8444648749557658 and parameters: {'n_estimators_nuisance': 46, 'max_depth_nuisance': 7}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:10,517] Trial 38 finished with value: 0.8492130625287463 and parameters: {'n_estimators_nuisance': 116, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:13,340] Trial 39 finished with value: 0.8512000268737807 and parameters: {'n_estimators_nuisance': 88, 'max_depth_nuisance': 7}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:14,607] Trial 40 finished with value: 0.8412070967330226 and parameters: {'n_estimators_nuisance': 32, 'max_depth_nuisance': 25}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:15,495] Trial 41 finished with value: 0.8387694115542171 and parameters: {'n_estimators_nuisance': 98, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:16,771] Trial 42 finished with value: 0.8409479350483986 and parameters: {'n_estimators_nuisance': 82, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:18,092] Trial 43 finished with value: 0.8401245743788259 and parameters: {'n_estimators_nuisance': 153, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:19,844] Trial 44 finished with value: 0.8434756284994833 and parameters: {'n_estimators_nuisance': 65, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:21,762] Trial 45 finished with value: 0.8451115987997999 and parameters: {'n_estimators_nuisance': 50, 'max_depth_nuisance': 8}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:23,522] Trial 46 finished with value: 0.842108214109248 and parameters: {'n_estimators_nuisance': 120, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:24,146] Trial 47 finished with value: 0.8492818670628405 and parameters: {'n_estimators_nuisance': 14, 'max_depth_nuisance': 29}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:29,971] Trial 48 finished with value: 0.8604042851883956 and parameters: {'n_estimators_nuisance': 182, 'max_depth_nuisance': 10}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:32,808] Trial 49 finished with value: 0.8491458727250417 and parameters: {'n_estimators_nuisance': 97, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:35,393] Trial 50 finished with value: 0.8478549784624454 and parameters: {'n_estimators_nuisance': 73, 'max_depth_nuisance': 22}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:36,311] Trial 51 finished with value: 0.8387476993794126 and parameters: {'n_estimators_nuisance': 100, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:37,300] Trial 52 finished with value: 0.8387486142854308 and parameters: {'n_estimators_nuisance': 110, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:41,125] Trial 53 finished with value: 0.8533845974011482 and parameters: {'n_estimators_nuisance': 133, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:41,815] Trial 54 finished with value: 0.8369904019034808 and parameters: {'n_estimators_nuisance': 37, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:43,186] Trial 55 finished with value: 0.8428916869666491 and parameters: {'n_estimators_nuisance': 34, 'max_depth_nuisance': 7}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:43,592] Trial 56 finished with value: 0.8417454482453965 and parameters: {'n_estimators_nuisance': 20, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:44,999] Trial 57 finished with value: 0.8402545366608952 and parameters: {'n_estimators_nuisance': 45, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:45,359] Trial 58 finished with value: 0.8814443931725696 and parameters: {'n_estimators_nuisance': 7, 'max_depth_nuisance': 9}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:47,456] Trial 59 finished with value: 0.8447037819013968 and parameters: {'n_estimators_nuisance': 55, 'max_depth_nuisance': 15}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:48,540] Trial 60 finished with value: 0.8395442258948982 and parameters: {'n_estimators_nuisance': 66, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:49,393] Trial 61 finished with value: 0.8385104671593674 and parameters: {'n_estimators_nuisance': 92, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:50,152] Trial 62 finished with value: 0.8379313466958876 and parameters: {'n_estimators_nuisance': 76, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:51,521] Trial 63 finished with value: 0.8413654037157335 and parameters: {'n_estimators_nuisance': 92, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:54,053] Trial 64 finished with value: 0.8471370283413241 and parameters: {'n_estimators_nuisance': 80, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:54,479] Trial 65 finished with value: 0.8374294906926323 and parameters: {'n_estimators_nuisance': 40, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:55,625] Trial 66 finished with value: 0.8388005691024599 and parameters: {'n_estimators_nuisance': 36, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:57,417] Trial 67 finished with value: 0.8449630098230781 and parameters: {'n_estimators_nuisance': 49, 'max_depth_nuisance': 7}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:59,093] Trial 68 finished with value: 0.8423566325073685 and parameters: {'n_estimators_nuisance': 41, 'max_depth_nuisance': 18}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:29:59,791] Trial 69 finished with value: 0.8432102857812726 and parameters: {'n_estimators_nuisance': 18, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:00,855] Trial 70 finished with value: 0.8395578695080244 and parameters: {'n_estimators_nuisance': 63, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:01,595] Trial 71 finished with value: 0.8379313466958876 and parameters: {'n_estimators_nuisance': 76, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:02,297] Trial 72 finished with value: 0.8375581069663144 and parameters: {'n_estimators_nuisance': 71, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:03,014] Trial 73 finished with value: 0.8376839400463968 and parameters: {'n_estimators_nuisance': 74, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:03,934] Trial 74 finished with value: 0.8387462300552294 and parameters: {'n_estimators_nuisance': 54, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:04,984] Trial 75 finished with value: 0.8384028389894236 and parameters: {'n_estimators_nuisance': 32, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:05,703] Trial 76 finished with value: 0.8375581069663144 and parameters: {'n_estimators_nuisance': 71, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:07,370] Trial 77 finished with value: 0.8429544705290505 and parameters: {'n_estimators_nuisance': 62, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:07,812] Trial 78 finished with value: 0.8372489323689495 and parameters: {'n_estimators_nuisance': 42, 'max_depth_nuisance': 3}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:08,760] Trial 79 finished with value: 0.8403292264180668 and parameters: {'n_estimators_nuisance': 25, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:09,534] Trial 80 finished with value: 0.8371308301582224 and parameters: {'n_estimators_nuisance': 41, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:10,232] Trial 81 finished with value: 0.83705019670631 and parameters: {'n_estimators_nuisance': 39, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:10,946] Trial 82 finished with value: 0.8370168846903381 and parameters: {'n_estimators_nuisance': 40, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:11,681] Trial 83 finished with value: 0.8370168846903381 and parameters: {'n_estimators_nuisance': 40, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:12,203] Trial 84 finished with value: 0.8375853613346683 and parameters: {'n_estimators_nuisance': 28, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:13,955] Trial 85 finished with value: 0.8442562603533649 and parameters: {'n_estimators_nuisance': 46, 'max_depth_nuisance': 8}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:14,452] Trial 86 finished with value: 0.8475976048409527 and parameters: {'n_estimators_nuisance': 14, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:15,411] Trial 87 finished with value: 0.8388348619310421 and parameters: {'n_estimators_nuisance': 55, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:16,864] Trial 88 finished with value: 0.8435553941652268 and parameters: {'n_estimators_nuisance': 38, 'max_depth_nuisance': 7}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:17,592] Trial 89 finished with value: 0.8402209945083552 and parameters: {'n_estimators_nuisance': 21, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:18,777] Trial 90 finished with value: 0.8408201505351982 and parameters: {'n_estimators_nuisance': 29, 'max_depth_nuisance': 11}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:19,576] Trial 91 finished with value: 0.837368394539858 and parameters: {'n_estimators_nuisance': 43, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:20,537] Trial 92 finished with value: 0.8385741722244079 and parameters: {'n_estimators_nuisance': 52, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:22,203] Trial 93 finished with value: 0.8430942414991943 and parameters: {'n_estimators_nuisance': 46, 'max_depth_nuisance': 6}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:22,876] Trial 94 finished with value: 0.8370020907012593 and parameters: {'n_estimators_nuisance': 36, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:24,189] Trial 95 finished with value: 0.843139897345287 and parameters: {'n_estimators_nuisance': 35, 'max_depth_nuisance': 7}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:25,851] Trial 96 finished with value: 0.842614606566614 and parameters: {'n_estimators_nuisance': 59, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:26,447] Trial 97 finished with value: 0.8370307355401991 and parameters: {'n_estimators_nuisance': 32, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:26,673] Trial 98 finished with value: 0.9018523922374757 and parameters: {'n_estimators_nuisance': 5, 'max_depth_nuisance': 5}. Best is trial 11 with value: 0.8369717349425706.
[I 2024-06-24 13:30:26,987] Trial 99 finished with value: 0.851871097086791 and parameters: {'n_estimators_nuisance': 14, 'max_depth_nuisance': 4}. Best is trial 11 with value: 0.8369717349425706.

Optimizing over architectures¶

optuna's flexibility allows for not only the search over classical hyperparameters of a given estimator but also to iterate over the choice of base estimator architectures. Pushing it a step further, one can even optimize over the space of MetaLearner architectures.

In the following example we will attempt to optimize over the following search space:

1. MetaLearner: R-Learner vs DR-Learner
2. Nuisance model: LGBMRegressor vs Ridge
3. Hyperparameter: n_estimators if LGBMRegressor and alpha if Ridge

Note that the choice of the base learner in the second step should be conditioned on the choice in the first step. In other words, we do not want to update our belief system on outcome learners for the R-Learner by observing outcome learner for the DR-Learner. The same idea applies to the interplay between steps two and three. This conditioning becomes apparent in the source code below via the underscores, e.g. nuisance_r, which is only sampled (and thereby updated) if we are using an R-Learner and and nuisance_dr, which is only sampled (and thereby updated) if we are using a DR-Learner.

In [5]:

import optuna
from metalearners.utils import metalearner_factory, simplify_output
from sklearn.linear_model import Ridge
from metalearners.rlearner import r_loss


# Arbitrary models for R-Loss
outcome_estimates = (
    LGBMRegressor(verbose=-1).fit(X_train, y_train).predict(X_validation)
)
propensity_scores = (
    LGBMClassifier(verbose=-1).fit(X_train, w_train).predict(X_validation)
)


def objective_overall(trial):

    ### SAMPLING

    # Highest level of granularity: we sample the MetaLearner architecture.
    architecture = trial.suggest_categorical("architecture", ["R", "DR"])

    # We distinguish cases because we do not want a DR-Learner run to influence
    # the optimizing process of R-Learner-related parameters.
    if architecture == "R":

        # Second level of granularity: we sample the nuisance base model.
        # Note that this is conditioned on using the R-Learner.
        nuisance_r = trial.suggest_categorical("nuisance_r", ["LGBMRegressor", "Ridge"])
        nuisance_dr = None
        nuisance_dr_lin_reg_alpha = None
        nuisance_dr_lgbm_n_estimators = None

        if nuisance_r == "LGBMRegressor":

            # Lowest level of granularity: we sample the nuisance base model hyperparameters.
            nuisance_r_lgbm_n_estimators = trial.suggest_int(
                "nuisance_r_lgbm_n_estimators", 5, 250
            )
            nuisance_r_lin_reg_alpha = None

            nuisance_params = {
                "n_estimators": nuisance_r_lgbm_n_estimators,
                "verbose": -1,
            }
        else:
            nuisance_r_lin_reg_alpha = trial.suggest_float(
                "nuisance_r_lin_reg_alpha", 0, 10
            )
            nuisance_r_lgbm_n_estimators = None

            nuisance_params = {"alpha": nuisance_r_lin_reg_alpha}

    else:
        nuisance_dr = trial.suggest_categorical(
            "nuisance_dr", ["LGBMRegressor", "Ridge"]
        )
        nuisance_r = None
        nuisance_r_lin_reg_alpha = None
        nuisance_r_lgbm_n_estimators = None

        if nuisance_dr == "LGBMRegressor":

            # Lowest level of granularity: we sample the nuisance base model hyperparameters.
            nuisance_dr_lgbm_n_estimators = trial.suggest_int(
                "nuisance_dr_lgbm_n_estimators", 5, 250
            )
            nuisance_dr_lin_reg_alpha = None

            nuisance_params = {
                "n_estimators": nuisance_dr_lgbm_n_estimators,
                "verbose": -1,
            }

        else:
            nuisance_dr_lin_reg_alpha = trial.suggest_float(
                "nuisance_dr_lin_reg_alpha", 0, 10
            )
            nuisance_dr_lgbm_n_estimators = None

            nuisance_params = {"alpha": nuisance_dr_lin_reg_alpha}

    ### LEARNING

    _metalearner_factory = metalearner_factory(architecture)
    # We know that only one of them is not None, therefore we can use or.
    nuisance_model_type = nuisance_r or nuisance_dr
    metalearner = _metalearner_factory(
        nuisance_model_factory=(
            LGBMRegressor if nuisance_model_type == "LGBMRegressor" else Ridge
        ),
        nuisance_model_params=nuisance_params,
        propensity_model_factory=LGBMClassifier,
        propensity_model_params={"n_estimators": 5, "max_depth": 5, "verbose": -1},
        treatment_model_factory=LGBMRegressor,
        treatment_model_params={"n_estimators": 5, "max_depth": 5, "verbose": -1},
        is_classification=False,
        n_variants=2,
    )
    metalearner.fit(X_train, y_train, w_train)

    ### EVALUATING

    cate_estimates = simplify_output(metalearner.predict(X_validation, is_oos=True))

    return r_loss(
        cate_estimates=cate_estimates,
        outcome_estimates=outcome_estimates,
        propensity_scores=propensity_scores,
        outcomes=y_validation,
        treatments=w_validation,
    )


study_overall = optuna.create_study(direction="minimize")
study_overall.optimize(objective_overall, n_trials=100)

import optuna from metalearners.utils import metalearner_factory, simplify_output from sklearn.linear_model import Ridge from metalearners.rlearner import r_loss # Arbitrary models for R-Loss outcome_estimates = ( LGBMRegressor(verbose=-1).fit(X_train, y_train).predict(X_validation) ) propensity_scores = ( LGBMClassifier(verbose=-1).fit(X_train, w_train).predict(X_validation) ) def objective_overall(trial): ### SAMPLING # Highest level of granularity: we sample the MetaLearner architecture. architecture = trial.suggest_categorical("architecture", ["R", "DR"]) # We distinguish cases because we do not want a DR-Learner run to influence # the optimizing process of R-Learner-related parameters. if architecture == "R": # Second level of granularity: we sample the nuisance base model. # Note that this is conditioned on using the R-Learner. nuisance_r = trial.suggest_categorical("nuisance_r", ["LGBMRegressor", "Ridge"]) nuisance_dr = None nuisance_dr_lin_reg_alpha = None nuisance_dr_lgbm_n_estimators = None if nuisance_r == "LGBMRegressor": # Lowest level of granularity: we sample the nuisance base model hyperparameters. nuisance_r_lgbm_n_estimators = trial.suggest_int( "nuisance_r_lgbm_n_estimators", 5, 250 ) nuisance_r_lin_reg_alpha = None nuisance_params = { "n_estimators": nuisance_r_lgbm_n_estimators, "verbose": -1, } else: nuisance_r_lin_reg_alpha = trial.suggest_float( "nuisance_r_lin_reg_alpha", 0, 10 ) nuisance_r_lgbm_n_estimators = None nuisance_params = {"alpha": nuisance_r_lin_reg_alpha} else: nuisance_dr = trial.suggest_categorical( "nuisance_dr", ["LGBMRegressor", "Ridge"] ) nuisance_r = None nuisance_r_lin_reg_alpha = None nuisance_r_lgbm_n_estimators = None if nuisance_dr == "LGBMRegressor": # Lowest level of granularity: we sample the nuisance base model hyperparameters. nuisance_dr_lgbm_n_estimators = trial.suggest_int( "nuisance_dr_lgbm_n_estimators", 5, 250 ) nuisance_dr_lin_reg_alpha = None nuisance_params = { "n_estimators": nuisance_dr_lgbm_n_estimators, "verbose": -1, } else: nuisance_dr_lin_reg_alpha = trial.suggest_float( "nuisance_dr_lin_reg_alpha", 0, 10 ) nuisance_dr_lgbm_n_estimators = None nuisance_params = {"alpha": nuisance_dr_lin_reg_alpha} ### LEARNING _metalearner_factory = metalearner_factory(architecture) # We know that only one of them is not None, therefore we can use or. nuisance_model_type = nuisance_r or nuisance_dr metalearner = _metalearner_factory( nuisance_model_factory=( LGBMRegressor if nuisance_model_type == "LGBMRegressor" else Ridge ), nuisance_model_params=nuisance_params, propensity_model_factory=LGBMClassifier, propensity_model_params={"n_estimators": 5, "max_depth": 5, "verbose": -1}, treatment_model_factory=LGBMRegressor, treatment_model_params={"n_estimators": 5, "max_depth": 5, "verbose": -1}, is_classification=False, n_variants=2, ) metalearner.fit(X_train, y_train, w_train) ### EVALUATING cate_estimates = simplify_output(metalearner.predict(X_validation, is_oos=True)) return r_loss( cate_estimates=cate_estimates, outcome_estimates=outcome_estimates, propensity_scores=propensity_scores, outcomes=y_validation, treatments=w_validation, ) study_overall = optuna.create_study(direction="minimize") study_overall.optimize(objective_overall, n_trials=100)

[I 2024-06-24 13:30:27,597] A new study created in memory with name: no-name-0752a341-284a-424f-9e2e-398ed679066b
[I 2024-06-24 13:30:34,032] Trial 0 finished with value: 0.8365666989350328 and parameters: {'architecture': 'R', 'nuisance_r': 'LGBMRegressor', 'nuisance_r_lgbm_n_estimators': 179}. Best is trial 0 with value: 0.8365666989350328.
[I 2024-06-24 13:30:34,508] Trial 1 finished with value: 0.8369255991554195 and parameters: {'architecture': 'R', 'nuisance_r': 'Ridge', 'nuisance_r_lin_reg_alpha': 4.542475752771536}. Best is trial 0 with value: 0.8365666989350328.
[I 2024-06-24 13:30:39,029] Trial 2 finished with value: 0.8358899088075391 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 54}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:43,603] Trial 3 finished with value: 0.8364125332969841 and parameters: {'architecture': 'R', 'nuisance_r': 'LGBMRegressor', 'nuisance_r_lgbm_n_estimators': 114}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:44,056] Trial 4 finished with value: 0.8368104376355334 and parameters: {'architecture': 'R', 'nuisance_r': 'Ridge', 'nuisance_r_lin_reg_alpha': 0.4002537648211202}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:44,570] Trial 5 finished with value: 0.8360189773162245 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 5.094443360435826}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:45,055] Trial 6 finished with value: 0.836170897517627 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 6.938444059357295}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:45,515] Trial 7 finished with value: 0.8366658519401137 and parameters: {'architecture': 'R', 'nuisance_r': 'Ridge', 'nuisance_r_lin_reg_alpha': 1.2804383906003336}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:57,864] Trial 8 finished with value: 0.8360220025481729 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 187}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:58,330] Trial 9 finished with value: 0.8364688376594042 and parameters: {'architecture': 'R', 'nuisance_r': 'Ridge', 'nuisance_r_lin_reg_alpha': 6.526099625721042}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:30:59,670] Trial 10 finished with value: 0.8377621892834751 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 10}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:31:00,184] Trial 11 finished with value: 0.836806012475984 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 1.5959907714281774}. Best is trial 2 with value: 0.8358899088075391.
[I 2024-06-24 13:31:00,669] Trial 12 finished with value: 0.8357196871278362 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 4.382493256163982}. Best is trial 12 with value: 0.8357196871278362.
[I 2024-06-24 13:31:05,085] Trial 13 finished with value: 0.8359352584971095 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 55}. Best is trial 12 with value: 0.8357196871278362.
[I 2024-06-24 13:31:12,740] Trial 14 finished with value: 0.8352000427902841 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 109}. Best is trial 14 with value: 0.8352000427902841.
[I 2024-06-24 13:31:13,222] Trial 15 finished with value: 0.8362926499975202 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 1.2517341626247007}. Best is trial 14 with value: 0.8352000427902841.
[I 2024-06-24 13:31:23,607] Trial 16 finished with value: 0.8355640135291108 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 155}. Best is trial 14 with value: 0.8352000427902841.
[I 2024-06-24 13:31:33,860] Trial 17 finished with value: 0.8354098424370239 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 150}. Best is trial 14 with value: 0.8352000427902841.
[I 2024-06-24 13:31:49,158] Trial 18 finished with value: 0.8342358030537382 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 241}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:32:03,562] Trial 19 finished with value: 0.8345946060089113 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 222}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:32:18,915] Trial 20 finished with value: 0.8347326139723813 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 242}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:32:34,868] Trial 21 finished with value: 0.83588652016062 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 250}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:32:50,562] Trial 22 finished with value: 0.8350426983577895 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 247}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:33:04,787] Trial 23 finished with value: 0.8351456266951328 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 216}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:33:18,022] Trial 24 finished with value: 0.8351859218028067 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 206}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:33:32,411] Trial 25 finished with value: 0.8348826146466184 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 223}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:33:44,453] Trial 26 finished with value: 0.8360629026210906 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 181}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:34:00,451] Trial 27 finished with value: 0.8348044268302125 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 250}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:34:15,380] Trial 28 finished with value: 0.834989746240632 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 225}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:34:16,657] Trial 29 finished with value: 0.8366817243330795 and parameters: {'architecture': 'R', 'nuisance_r': 'LGBMRegressor', 'nuisance_r_lgbm_n_estimators': 19}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:34:29,340] Trial 30 finished with value: 0.8352184383627659 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 192}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:34:45,124] Trial 31 finished with value: 0.8350579599909539 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 248}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:35:00,013] Trial 32 finished with value: 0.8350634945778808 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 229}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:35:15,802] Trial 33 finished with value: 0.8355419588890591 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 250}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:35:23,938] Trial 34 finished with value: 0.8361194553157467 and parameters: {'architecture': 'R', 'nuisance_r': 'LGBMRegressor', 'nuisance_r_lgbm_n_estimators': 245}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:35:38,040] Trial 35 finished with value: 0.8349977243237731 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 207}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:35:52,629] Trial 36 finished with value: 0.8349057183387861 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 227}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:35:53,093] Trial 37 finished with value: 0.8374235329621648 and parameters: {'architecture': 'R', 'nuisance_r': 'Ridge', 'nuisance_r_lin_reg_alpha': 9.849083510512918}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:36:03,659] Trial 38 finished with value: 0.8353151329974942 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 158}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:36:11,553] Trial 39 finished with value: 0.8359527209047248 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 110}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:36:12,400] Trial 40 finished with value: 0.8378260256069935 and parameters: {'architecture': 'R', 'nuisance_r': 'LGBMRegressor', 'nuisance_r_lgbm_n_estimators': 8}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:36:27,064] Trial 41 finished with value: 0.8357427540880243 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 227}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:36:42,062] Trial 42 finished with value: 0.8350604103377771 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 232}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:36:55,503] Trial 43 finished with value: 0.835313525384969 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 203}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:36:55,987] Trial 44 finished with value: 0.8363999834685643 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 9.153277206854481}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:37:11,132] Trial 45 finished with value: 0.8352094641913799 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 236}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:37:25,612] Trial 46 finished with value: 0.8347375708250164 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 216}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:37:26,097] Trial 47 finished with value: 0.8364518714108701 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 9.648516610564526}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:37:26,558] Trial 48 finished with value: 0.836608356178382 and parameters: {'architecture': 'R', 'nuisance_r': 'Ridge', 'nuisance_r_lin_reg_alpha': 9.7956015863694}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:37:38,103] Trial 49 finished with value: 0.8357644020771988 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 173}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:37:51,764] Trial 50 finished with value: 0.8356344185660193 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 209}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:38:05,960] Trial 51 finished with value: 0.8349566570245126 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 219}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:38:21,096] Trial 52 finished with value: 0.8343999043950624 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 239}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:38:36,210] Trial 53 finished with value: 0.8350164711086289 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 239}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:38:36,684] Trial 54 finished with value: 0.836892877736011 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 0.18380748954816362}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:38:49,311] Trial 55 finished with value: 0.8349287983532163 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 196}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:39:04,259] Trial 56 finished with value: 0.8345171405712465 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 235}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:39:05,227] Trial 57 finished with value: 0.837278875998113 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 5}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:39:05,716] Trial 58 finished with value: 0.8367195615762762 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 3.873052787622435}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:39:10,360] Trial 59 finished with value: 0.8360197281295247 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 57}. Best is trial 18 with value: 0.8342358030537382.
[I 2024-06-24 13:39:25,426] Trial 60 finished with value: 0.8342052997834747 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 238}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:39:40,698] Trial 61 finished with value: 0.8352420380395968 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 238}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:39:55,977] Trial 62 finished with value: 0.8345683646878848 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 236}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:40:11,346] Trial 63 finished with value: 0.8345841112267551 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 238}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:40:26,567] Trial 64 finished with value: 0.8355014360968269 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 236}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:40:32,041] Trial 65 finished with value: 0.835402917218263 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 74}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:40:46,589] Trial 66 finished with value: 0.8352468104401285 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 214}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:40:50,508] Trial 67 finished with value: 0.8359967379909685 and parameters: {'architecture': 'R', 'nuisance_r': 'LGBMRegressor', 'nuisance_r_lgbm_n_estimators': 98}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:41:05,896] Trial 68 finished with value: 0.834581314441489 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 237}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:41:21,231] Trial 69 finished with value: 0.8343917423087528 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 239}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:41:24,122] Trial 70 finished with value: 0.8361906598239933 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 31}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:41:39,024] Trial 71 finished with value: 0.8347442641049424 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 237}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:41:54,266] Trial 72 finished with value: 0.8350849854591442 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 241}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:42:08,780] Trial 73 finished with value: 0.8350643357956492 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 223}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:42:24,296] Trial 74 finished with value: 0.8350786736190959 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 242}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:42:39,157] Trial 75 finished with value: 0.8346586773235721 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 229}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:42:52,407] Trial 76 finished with value: 0.8348753973981645 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 202}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:43:08,391] Trial 77 finished with value: 0.8352334546262877 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 249}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:43:08,973] Trial 78 finished with value: 0.8366295510610248 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 6.872566417649445}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:43:09,428] Trial 79 finished with value: 0.836510635780872 and parameters: {'architecture': 'R', 'nuisance_r': 'Ridge', 'nuisance_r_lin_reg_alpha': 4.342523862892199}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:43:23,541] Trial 80 finished with value: 0.8348657415346825 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 216}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:43:38,281] Trial 81 finished with value: 0.8348190023558525 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 231}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:43:54,132] Trial 82 finished with value: 0.8347489871849719 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 243}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:44:08,407] Trial 83 finished with value: 0.8351330177990129 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 225}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:44:17,903] Trial 84 finished with value: 0.8348917265474182 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 135}. Best is trial 60 with value: 0.8342052997834747.
[I 2024-06-24 13:44:33,912] Trial 85 finished with value: 0.8341679096865403 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 250}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:44:50,004] Trial 86 finished with value: 0.8350793370178582 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 248}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:45:04,808] Trial 87 finished with value: 0.8346638007073203 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 232}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:45:20,711] Trial 88 finished with value: 0.8353733471331339 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 250}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:45:36,018] Trial 89 finished with value: 0.8345094557607371 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 241}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:45:50,035] Trial 90 finished with value: 0.8350057212001983 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 220}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:46:05,552] Trial 91 finished with value: 0.8344050056655915 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 241}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:46:21,078] Trial 92 finished with value: 0.8348676785979172 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 243}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:46:36,155] Trial 93 finished with value: 0.8344743459000354 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 232}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:46:51,140] Trial 94 finished with value: 0.835315048867249 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 230}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:47:05,914] Trial 95 finished with value: 0.8348480840288921 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 224}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:47:14,425] Trial 96 finished with value: 0.8358097919152879 and parameters: {'architecture': 'R', 'nuisance_r': 'LGBMRegressor', 'nuisance_r_lgbm_n_estimators': 250}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:47:28,259] Trial 97 finished with value: 0.834241411709681 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 209}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:47:28,743] Trial 98 finished with value: 0.8365182044311655 and parameters: {'architecture': 'DR', 'nuisance_dr': 'Ridge', 'nuisance_dr_lin_reg_alpha': 7.478356878487755}. Best is trial 85 with value: 0.8341679096865403.
[I 2024-06-24 13:47:42,675] Trial 99 finished with value: 0.834996319731582 and parameters: {'architecture': 'DR', 'nuisance_dr': 'LGBMRegressor', 'nuisance_dr_lgbm_n_estimators': 213}. Best is trial 85 with value: 0.8341679096865403.