<commit_before># Examples with OBD
Here, we use the dataset and pipeline to implement and evaluate OPE.
We first evaluate well-known off-policy estimators with the ground-truth performance of a counterfactual policy.
We then select and use such an estimator to improve the platform’s fashion item recommendation.

## Additional Implementations

- [`dataset.py`](./dataset.py):
    We implement two ways of engeering original features.
    This includes **CONTEXT SET 1** (user features only, such as age, gender, and length of  membership) and **CONTEXT SET 2** (Context Set 1 plus user-item affinity induced by the number of past clicks observed between each user-item pair)

- [`logistic_bandit.py`](./logistic_bandit.py):
    We implement three logistic contextual bandit algorithms *logistic epsilon-greedy (logistic EGreedy)*, *logistic upper confidence bound (logistic UCB)*, and *logistic thompson sampling (logistic TS)*.
    They construct our counterfactual policy search space.
    Note that we modify the contextual bandit algorithms to adjust to ZOZOTOWN top-3 recommendation setting.
    For example, modified logistic TS selects three actions with the three highest sampled rewards.

## Running experiments

**Step 1. Evaluating a regression model**

We evaluate LightGBM as a regression model to be used in DM and DR.

```bash
python eval_regression_model.py\
    --n_splits 1\ # the number of train-test splits
    --behavior_policy bts\ # a behavior policy
    --campaign men # a campaign

# click prediction performance of a regression model
# =========================
# AUC: 0.615171
# RCE: 0.010273
# =========================
```

**Step 2. Selecting Off-policy Estimator** (Section 5.1)

We select the best off-policy estimator among DM, IPW, and DR.

```bash
python off_policy_estimator_selection.py\
    --n_splits 1\
    --n_estimators 10\ # the number of bagged samples used in bagging aggregation
    --behavior_policy bts\
    --campaign men

# relative estiamtion erros of three OPE estimators
# =========================
# random_state=0
# -----
# DM: 0.342532
# IPW: 0.684355
# DR: 1.035945
# =========================
```


**Step 3. Selecting Counterfactual Policy** (Section 5.2)

We evaluate the performance of counterfactual policies based on logistic contextual bandit with OPE estimators.

```bash
python cf_policy_selection.py\
    --n_splits 1\
    --n_estimators 10\
    --context_set 1\ # a used context set
    --counterfactual_policy logistic_ucb\ # a used counterfactual bandit algorithm
    --epsilon 0.1\ # an exploration hyperparameter
    --behavior_policy bts\
    --campaign men

# estimated policy values relative to the behavior policy (the Bernoulli TS here) of a counterfactual policy (the Logistic UCB with Context Set 1 here) by three OPE estimators
# =========================
# random_state=0
# -----
# DM: 0.826714
# IPW: 0.922185
# DR: 1.013117
# =========================
```


## References
1. <NAME> and <NAME>. The offset tree for learning with partial labels. In *Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining*, pages 129–138, 2009.

1. <NAME> and <NAME>. An empirical evaluation of thompson sampling. In *Advances in neural information processing systems*, pages 2249–2257, 2011.

1. <NAME>, <NAME>, <NAME>, and <NAME>. Doubly Robust Policy Evaluation and Optimization. *Statistical Science*, 29:485–511, 2014.

1. <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, and <NAME>. Lightgbm: A highly efficient gradient boosting decision tree. In *Advances in Neural Information Processing Systems*, pages 3146–3154, 2017.

1. <NAME>, <NAME>, <NAME>, and <NAME>. A Contextual-bandit Approach to Personalized News Article Recommendation. In *Proceedings of the 19th International Conference on World Wide Web*, pages 661–670. ACM, 2010.

1. <NAME>, <NAME>, <NAME>, and <NAME>. Learning from Logged Implicit Exploration Data. In Advances in *Neural Information Processing Systems*, pages 2217–2225, 2010.<commit_msg>update examples/obd/README<commit_after># Examples with OBD
Here, we use the dataset and pipeline to implement and evaluate OPE.
We first evaluate well-known off-policy estimators with the ground-truth performance of a counterfactual policy.
We then select and use such an estimator to improve the platform’s fashion item recommendation.

## Additional Implementations

- [`dataset.py`](./dataset.py):
    We implement two ways of engeering original features.
    This includes **CONTEXT SET 1** (user features only, such as age, gender, and length of  membership) and **CONTEXT SET 2** (Context Set 1 plus user-item affinity induced by the number of past clicks observed between each user-item pair).

## Running experiments

**Step 1. Evaluating a regression model**

We evaluate LightGBM as a regression model to be used in DM and DR.

```bash
python eval_regression_model.py\
    --n_splits 1\ # the number of train-test splits
    --behavior_policy bts\ # a behavior policy
    --campaign men # a campaign

# click prediction performance of a regression model
# =========================
# AUC: 0.615171
# RCE: 0.010273
# =========================
```

**Step 2. Selecting Off-policy Estimator** (Section 5.1)

We select the best off-policy estimator among DM, IPW, and DR.

```bash
python off_policy_estimator_selection.py\
    --n_splits 1\
    --n_estimators 10\ # the number of bagged samples used in bagging aggregation
    --behavior_policy bts\
    --campaign men

# relative estiamtion erros of three OPE estimators
# =========================
# random_state=0
# -----
# DM: 0.342532
# IPW: 0.684355
# DR: 1.035945
# =========================
```


**Step 3. Selecting Counterfactual Policy** (Section 5.2)

We evaluate the performance of counterfactual policies based on logistic contextual bandit with OPE estimators.

```bash
python cf_policy_selection.py\
    --n_splits 1\
    --n_estimators 10\
    --context_set 1\ # a used context set
    --counterfactual_policy logistic_ucb\ # a used counterfactual bandit algorithm
    --epsilon 0.1\ # an exploration hyperparameter
    --behavior_policy bts\
    --campaign men

# estimated policy values relative to the behavior policy (the Bernoulli TS here) of a counterfactual policy (the Logistic UCB with Context Set 1 here) by three OPE estimators
# =========================
# random_state=0
# -----
# DM: 0.826714
# IPW: 0.922185
# DR: 1.013117
# =========================
```

{'fraction_non_alphanumeric': 0.13238697731831509, 'fraction_numerical': 0.0365684307977164, 'mean_word_length': 4.124110671936759, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4701602', 'n_tokens_mistral': 2068, 'n_tokens_neox': 1792, 'n_words': 774}