<jupyter_start><jupyter_text>Evaluation of the AL/IRL model In this section, we evaluate the performance of the expert policy generated by AL/IRL component, which saved for our simulationin policyV1.npy. We also use its associated states dataset stored in file statesV1.npy .<jupyter_code>max_episode = 500
slate_size = 2
num_candidates = 5

env_config = {
        'num_candidates': num_candidates,
        'slate_size': slate_size,
        'resample_documents': True,
        'seed': 0,
        'reward_function':clicked_engagement_reward
    }

# User simulation environment: interest evolution model presented in the paper of SlateQ
recsim_gym_env = interest_evolution.create_environment(env_config)

# Load the learned policy from the Expert IRL model as well as the resulted/associated states
states = load('../datasets_states/statesV1.npy', allow_pickle=True)
policy_ = load('../datasets_states/policyV1.npy', allow_pickle=True)

# Instanciate the Expert IRL agent 
agent_irl = InverseRLAgent(recsim_gym_env, states, policy_, num_cand=num_candidates,
                               slate_size=slate_size, max_steps_per_episode=100, ml_model=False)


results = []
# generate independent episodes
for i in range(max_episode):
    steps, reward, time_, found, exp_watch, user_metrics = agent_irl.run_one_episode()
    results += [[i,steps, reward, time_, found, exp_watch, user_metrics]]
    print("episode",i)
episodes = np.linspace(0,max_episode-1, max_episode)
user_reward = []
expert_reward = []
episode_steps = []
episode_found_state = []
episode_clicked = []
episode_videos_length = []
episode_total_quality = []
episode_expert_quality = []

for i in range(len(results)):
    episode_steps += [results[i][1]]
    user_reward += [results[i][2]]
    episode_found_state += [results[i][4]]
    expert_reward += [results[i][5]]
    episode_clicked += [results[i][6][0]]
    episode_videos_length += [results[i][6][1]]
    episode_expert_quality += [results[i][6][2]]
    episode_total_quality += [results[i][6][3]]
    
ratio_watch = [user_reward[i] for i in range(len(episode_videos_length))]
from numpy import save
a = results
save('../results.npy', a)<jupyter_output><empty_output><jupyter_text>The table below shows the following simulation metrics:    episode_steps: length of the episode ( number of states)      episode_match_state: number of user states that have been matched by        the classification algorithm to similar expert states     T_clicked videos: Total number of videos that have been clicked by users within the episode (session)     T_episode_quality: total quality calculated for the episode     expert_deleg_quality: quality calculated for watched videos that belongs to states of episode_match_state    T_videos_length: duration of all videos watched in a given episode     T_watching_time: Total watching time of clicked videos    Note 1: In some episodes, T_watching_time values are greater than T_videos_length while it is supposed to be the contrary. However, this situation may happen because of some videos that are watched more than one time.Note 2: Metrics T_episode_quality and expert_deleg_quality gives an indea about how much it is beneficial to follow expert rated videos, which are recommended based on our approach. For instance, in the episode 0, we have 30 videos that have been seen through the expert policy of our approach. These videos are delivering a quality of 0.449174. However, for the rest 32 videos (62-30=32), which are watched by a random recommendation because their states are not matched to any expert state from the states dataset, they then led to dicrease the quality (by negative quality values) ending up by a total quality of  0.32 .<jupyter_code>import pandas as pd
data = np.array([episode_steps, episode_found_state,  episode_clicked, 
            episode_total_quality, episode_expert_quality, episode_videos_length, user_reward, ratio_watch])
pd.DataFrame(data, columns=np.arange(max_episode), index=["#episode_steps", "#episode_match_state",  "T_clicked videos",
                             "T_episode_quality", "expert_deleg_quality", "T_videos_length", "T_watching_time"," ratio_watching"])<jupyter_output><empty_output><jupyter_text>Figure 1 below shows that the delegated expert quality values positively increase the total quality as explained in note 2 of cell 7.<jupyter_code>#plt.plot(episodes, ratio_watch, color='blue', label = "watching_time", marker='o')
plt.scatter(episodes, episode_total_quality, color='red',  s=5, label = "$Q_T$", marker='o')
plt.scatter(episodes, episode_expert_quality, color='green', s=5, label = "$Q_d$", marker='o')
plt.ylim(-1, 1)
#plt.xticks(np.arange(min(episodes), max(episodes)+1, 1.0))
plt.xlabel("episodes")
plt.ylabel("quality rate")
plt.legend(loc='upper right')
plt.title("Figure 1: expert delegated quality $Q_d$ and total quality $Q_T$ by episodes")
plt.savefig("q_eval.png", bbox_inches='tight')
plt.show()<jupyter_output><empty_output><jupyter_text>Comparison of our approach recFEBR  with the baseline methodes: recFSQ and recNaive  Training ...<jupyter_code>from recsim.agents import  full_slate_q_agent, random_agent
from irl_agent import InverseRLAgent
import tensorflow.compat.v1 as tf
import time
from numpy import load


def run_one_episode(env, agent, max_steps_per_episode=100):
    observation = env.reset()
    action = agent.begin_episode(observation)
    step_number = 0
    total_watch = 0.
    q_videos = []
    w_videos = []
    total_qual = 0
    start_time = time.time()
    total_length_videos = 0
    while True:
        observation, reward, done, info, _ = env.step(action)

        for j in range(len(observation['response'])):
            if observation['response'][j]['click'] == 1:
                index = action[j]
                total_length_videos += list(observation['doc'].values())[index][-1]
                total_watch += reward[1]
                total_qual += reward[0]
                q_videos += [reward[0]]
                w_videos += [reward[1]]

        # Update environment-specific metrics with responses to the slate.
        env.update_metrics(observation['response'], info)
        step_number += 1

        if done:
            break
        elif step_number == max_steps_per_episode:
            # Stop the run loop once we reach the true end of episode.
            break
        else:
            action = agent.step(reward[1], observation)

    agent.end_episode(reward[1], observation)
    time_diff = time.time() - start_time

    return step_number, total_watch, time_diff, total_qual/step_number, q_videos, w_videos
def clicked_quality_reward(responses):
    """Calculates the total clicked watchtime from a list of responses.

    Args:
      responses: A list of IEvResponse objects

    Returns:
      reward: A float representing the total watch time from the responses
    """
    qual = 0.0
    watch = 0.0
    for response in responses:
        if response.clicked:
            qual += float(response.quality)
            watch += float(response.watch_time)
    return [qual, watch]



def create_agent_random(slate_size, random_seed=0):
    action_space = spaces.MultiDiscrete(num_candidates * np.ones((slate_size,)))
    return random_agent.RandomAgent(action_space, random_seed)<jupyter_output><empty_output><jupyter_text>recNaive simulation<jupyter_code>slate_size = 2
num_candidates = 5
env_config1 = {
        'num_candidates': num_candidates,
        'slate_size': slate_size,
        'resample_documents': True,
        'seed': 0,
        'reward_function':clicked_quality_reward
    }

recsim_gym_env = interest_evolution.create_environment(env_config1)
agent = create_agent_random(slate_size)
results_r = []
for i in range(max_episode):
    steps_r, watch, time_r, q, q_vid, w_vid = run_one_episode(recsim_gym_env, agent)
    results_r += [[i,steps_r, watch, time_r, q, q_vid, w_vid ]]
    print("episode ", i)

    
episode_steps_r = []
episode_ratio_watch_r = []
episode_total_quality_r = []
episodes_qv = []
episodes_wv = []

for i in range(len(results_r)):
    episode_steps_r += [results_r[i][1]]
    episode_ratio_watch_r += [results_r[i][2]]
    episode_total_quality_r += [results_r[i][4]]
    episodes_qv += [results_r[i][5]]
    episodes_wv += [results_r[i][6]]<jupyter_output>episode  0
episode  1
episode  2
episode  3
episode  4
episode  5
episode  6
episode  7
episode  8
episode  9
episode  10
episode  11
episode  12
episode  13
episode  14
episode  15
episode  16
episode  17
episode  18
episode  19
episode  20
episode  21
episode  22
episode  23
episode  24
episode  25
episode  26
episode  27
episode  28
episode  29
episode  30
episode  31
episode  32
episode  33
episode  34
episode  35
episode  36
episode  37
episode  38
episode  39
episode  40
episode  41
episode  42
episode  43
episode  44
episode  45
episode  46
episode  47
episode  48
episode  49
episode  50
episode  51
episode  52
episode  53
episode  54
episode  55
episode  56
episode  57
episode  58
episode  59
episode  60
episode  61
episode  62
episode  63
episode  64
episode  65
episode  66
episode  67
episode  68
episode  69
episode  70
episode  71
episode  72
episode  73
episode  74
episode  75
episode  76
episode  77
episode  78
episode  79
episode  80
episode  81
episode  82
episode  83
ep[...]<jupyter_text>recFSQ simulation<jupyter_code>slate_size = 2
num_candidates = 5
env_config1 = {
        'num_candidates': num_candidates,
        'slate_size': slate_size,
        'resample_documents': True,
        'seed': 0,
        'reward_function':clicked_quality_reward
    }

recsim_gym_env = interest_evolution.create_environment(env_config1)
results_f = []

with tf.Session(config=tf.ConfigProto(allow_soft_placement=True)) as sess:
    
    agent = full_slate_q_agent.FullSlateQAgent(sess, recsim_gym_env.observation_space, recsim_gym_env.action_space)
    for i in range(max_episode):
        steps_f, watch, time_f, q, q_vid, w_vid = run_one_episode(recsim_gym_env, agent)
        results_f += [[i, steps_f, watch, time_f, q, q_vid, w_vid]]
        print("episode ", i)
        sess.run(tf.global_variables_initializer())

    
episode_steps_f = []
episode_ratio_watch_f = []
episode_total_quality_f = []
episodes_qvf = []
episodes_wvf = []


for i in range(len(results_f)):
    episode_steps_f += [results_f[i][1]]
    episode_ratio_watch_f += [results_f[i][2]]
    episode_total_quality_f += [results_f[i][4]]
    episodes_qvf += [results_f[i][5]]
    episodes_wvf += [results_f[i][6]]<jupyter_output>INFO:tensorflow:Creating FullSlateQAgent agent with the following parameters:<jupyter_text>Statistics and comparison<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

n_bins = 10

# Generate a normal distribution, center at x=0 and y=5

fig, axs = plt.subplots(1, 3, sharey=True, tight_layout=True)

# We can set the number of bins with the `bins` kwarg
axs[0].hist(episode_total_quality, bins=n_bins)
axs[0].set_title("recFEBR")
axs[1].hist(episode_total_quality_f, bins=n_bins)
axs[1].set_title("recFSQ")
axs[2].hist(episode_total_quality_r, bins=n_bins)
axs[2].set_title("revNaive")
st=fig.suptitle("Figure 2: Empirical distribution of total quality $Q_T$", fontsize="x-large")
st.set_y(1.1)
import pandas as pd
data_quality = [sum(episode_total_quality_f) / max_episode, sum(episode_total_quality_r) / max_episode, sum(episode_total_quality) / max_episode]
data_watch_time = [sum(episode_ratio_watch_f), sum(episode_ratio_watch_r), float(sum(user_reward))]
data = np.array([data_quality, data_watch_time])
pd.DataFrame(data, columns=["recFSQ", "recNaive", "recFEBR"], index=["Average total quality $Q_T$","Total watching time $W_T$"])
df_quality = pd.DataFrame([episode_total_quality, episode_total_quality_f, episode_total_quality_r], index=["FEBR","Full_Q","Naive_random_"])
data1 = df_quality.T
data1.describe()
name_file = "w_t_compar.png"
approaches = ["RecFSQ","RecNaive", "RecFEBR"]
plt.bar(approaches, data[1])
plt.ylabel('Total watch time (s)')
plt.title("Figure 3: Comparison of the total watch time $W_T$ (seconds)")
# saving file into result folder
#plt.savefig("../eval_results/"+name_file,, bbox_inches='tight')
name_file = "q_t_compar.png"
plt.bar(approaches, data[0])
plt.ylabel('Average total quality')
plt.yticks(np.arange(-0.2, 0.5, 0.05))
plt.title("Figure 4: Comparison of the average total quality $Q_T$ ")
# saving file into result folder
#plt.savefig("../eval_results/"+name_file, bbox_inches='tight')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0998798558269924, 'fraction_numerical': 0.02314777733279936, 'mean_word_length': 4.471516213847502, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '295967', 'license': 'MIT', 'n_tokens_mistral': 4573, 'n_tokens_neox': 4227, 'n_words': 1108, 'path': 'notebooks/models_eval.ipynb', 'repo_name': 'FEBR-rec/ExpertDrivenRec', 'size': '12485'}