<jupyter_start><jupyter_text>FrozenLakeS: 시작지점F: 얼음판H: 구멍(죽음)G: 골원하는대로 가지 않음 gym의 3가지 main method* reset() - reset environment to initial state, return first observation* render() - show current environment state (a more colorful version :) )* step(a) - commit action a and return (new observation, reward, is done, info)    * new observation - an observation right after commiting the action a    * reward - a number representing your reward for commiting action a    * is done - True if the MDP has just finished, False if still in progress    * info - some auxilary stuff about what just happened. Ignore it for now<jupyter_code>import gym

#create a single game instance
env = gym.make("FrozenLake-v0")
print("initial observation code:", env.reset())
print('printing observation:')
env.render()
print("observations:", env.observation_space, 'n=', env.observation_space.n)
print("actions:", env.action_space, 'n=', env.action_space.n)
env.action_space
print("taking action 2 (right)")
new_obs, reward, is_done, _ = env.step(2)
print("new observation code:", new_obs)
print("reward:", reward)
print("is game over?:", is_done)
print("printing new state:")
env.render()
action_to_i = {
    'left':0,
    'down':1,
    'right':2,
    'up':3
}<jupyter_output><empty_output><jupyter_text>Baseline: random search (2 points) Policy* 환경은 4x4 grid state를 가지고 각자 0~15의 index를 가진다.* 각 state마다 4가지 action이 존재(L,D,R,U)하며 각자 0~3의 index를 가짐.state가 주어지면 action을 취하는 agent의 policy를 정의해야한다. 지금은 state와 action이 작기 때문에 그냥 각 state마다 action을 저장하면 된다.<jupyter_code>import numpy as np
n_states = env.observation_space.n
n_actions = env.action_space.n
def get_random_policy():
    """
    Build a numpy array representing agent policy.
    This array must have one element per each of 16 environment states.
    Element must be an integer from 0 to 3, representing action
    to take from that state.
    """
    return np.random.randint(4,size=16)  #0~3

print(get_random_policy())
np.random.seed(1234)
policies = [get_random_policy() for i in range(10**4)]

assert all([len(p) == n_states for p in policies]), 'policy length should always be 16'
assert np.min(policies) == 0, 'minimal action id should be 0'
assert np.max(policies) == n_actions-1, 'maximal action id should match n_actions-1'

action_probas = np.unique(policies, return_counts=True)[-1] /10**4. /n_states
print("Action frequencies over 10^4 samples:",action_probas)
assert np.allclose(action_probas, [1. / n_actions] * n_actions, atol=0.05), "The policies aren't uniformly random (maybe it's just an extremely bad luck)"
print("Seems fine!")<jupyter_output>Action frequencies over 10^4 samples: [ 0.25014375  0.25130625  0.2495375   0.2490125 ]
Seems fine!<jupyter_text>Let's evaluate* game을 한번 실행하고, total reward를 return 하는 함수를 제작한다.<jupyter_code>def sample_reward(env, policy, t_max=100):
    """
    Interact with an environment, return sum of all rewards.
    If game doesn't end on t_max (e.g. agent walks into a wall), 
    force end the game and return whatever reward you got so far.
    Tip: see signature of env.step(...) method above.
    """
    s = env.reset()
    total_reward = 0
    is_done = False
    for _ in range(t_max):
        s, reward, is_done, info = env.step(policy[s])
        total_reward += reward
        if is_done:
            break
    return total_reward
print("generating 10^3 sessions...")
rewards = [sample_reward(env,get_random_policy()) for _ in range(10**3)]
assert all([type(r) in (int, float) for r in rewards]), 'sample_reward must return a single number'
assert all([0 <= r <= 1 for r in rewards]), 'total rewards should be between 0 and 1 for frozenlake (if solving taxi, delete this line)'
print("Looks good!")
def evaluate(policy, n_times=100):
    """Run several evaluations and average the score the policy gets."""
    rewards = [sample_reward(env, policy) for _ in range(n_times)]
    return float(np.mean(rewards))
def print_policy(policy):
    """a function that displays a policy in a human-readable way."""
    lake = "SFFFFHFHFFFHHFFG"
    assert env.spec.id == "FrozenLake-v0", "this function only works with frozenlake 4x4"

    
    # where to move from each tile (we're a bit unsure if this is accurate)
    arrows = ['>^v<'[a] for a in policy]
    
    #draw arrows above S and F only
    signs = [arrow if tile in "SF" else tile for arrow, tile in zip(arrows, lake)]
    
    for i in range(0, 16, 4):
        print(' '.join(signs[i:i+4]))

print("random policy:")
print_policy(get_random_policy())
best_policy = None
best_score = -float('inf')

from tqdm import tqdm
for i in tqdm(range(10000)):
    policy = get_random_policy()
    score = evaluate(policy)
    if score > best_score:
        best_score = score
        best_policy = policy
        print("New best score:", score)
        print("Best policy:")
        print_policy(best_policy)<jupyter_output>0%|          | 5/10000 [00:00<03:40, 45.42it/s]<jupyter_text>Part II 유전 알고리즘 (4점)이제 policy search를 행하기 위한 좀더 효율적인 방법을 고안할 것이다. crossover두 개의 정책이 존재할 때, 둘 중 어떤 것을 선택할지...<jupyter_code>def crossover(policy1, policy2, p=0.5):
    """
    for each state, with probability p take action from policy1, else policy2
    """
    policy = []
    for p1, p2 in zip(policy1, policy2):
        choice = np.random.uniform(0,1,1)
        if choice < p:
            policy.append(p1)
        else:
            policy.append(p2)
    return np.array(policy)<jupyter_output><empty_output><jupyter_text>mutation어떤 policy를 만들 때, 유전 변형을 일으킨다.<jupyter_code>def mutation(policy, p=0.1):
    """
    for each state, with probability p replace action with random action
    Tip: mutation can be written as crossover with random policy
    """
    new_policy = []
    for a in policy:
        choice = np.random.uniform(0,1,1)
        if choice >= p:
            new_policy.append(a)
        else:
            new_policy.append(np.random.randint(0,4))
    return np.array(new_policy)
np.random.seed(1234)
policies = [crossover(get_random_policy(), get_random_policy()) 
            for i in range(10**4)]

assert all([len(p) == n_states for p in policies]), 'policy length should always be 16'
assert np.min(policies) == 0, 'minimal action id should be 0'
assert np.max(policies) == n_actions-1, 'maximal action id should be n_actions-1'

assert any([np.mean(crossover(np.zeros(n_states), np.ones(n_states))) not in (0, 1)
               for _ in range(100)]), "Make sure your crossover changes each action independently"
print("Seems fine!")
n_epochs = 100 #how many cycles to make
pool_size = 100 #how many policies to maintain
n_crossovers = 50 #how many crossovers to make on each step
n_mutations = 50 #how many mutations to make on each tick
print("initializing...")
pool = [get_random_policy() for i in range(pool_size)]
pool_scores = [evaluate(policy) for policy in pool]#<evaluate every policy in the pool, return list of scores>
assert type(pool) == type(pool_scores) == list
assert len(pool) == len(pool_scores) == pool_size
assert all([type(score) in (float, int) for score in pool_scores])
#main loop
for epoch in range(n_epochs):
    print("Epoch %s:"%epoch)
    
    choices1 = [np.random.randint(len(pool)) for _ in range(n_crossovers)]
    choices2 = [np.random.randint(len(pool)) for _ in range(n_crossovers)]
    
    crossovered = [crossover(pool[c1], pool[c2]) for c1, c2 in zip(choices1, choices2)]
    pool = pool + crossovered
    mut = [np.random.randint(len(pool)) for _ in range(n_mutations)]
    mutated = [mutation(pool[m]) for m in mut]
    
    assert type(crossovered) == type(mutated) == list
    
    #add new policies to the pool
    pool = pool + mutated
    pool_scores = [evaluate(policy) for policy in pool]
    
    #select pool_size best policies
    selected_indices = np.argsort(pool_scores)[-pool_size:]
    pool = [pool[i] for i in selected_indices]
    pool_scores = [pool_scores[i] for i in selected_indices]

    #print the best policy so far (last in ascending score order)
    print("best score:", pool_scores[-1])
    print_policy(pool[-1])<jupyter_output>Epoch 0:
best score: 0.1
< v > ^
^ H v H
^ < ^ H
H < v G
Epoch 1:
best score: 0.22
> v ^ ^
> H > H
< > > H
H v v G
Epoch 2:
best score: 0.23
> v ^ ^
> H > H
< > > H
H v v G
Epoch 3:
best score: 0.22
> v ^ ^
> H > H
< > > H
H v v G
Epoch 4:
best score: 0.39
> < > >
> H > H
< ^ > H
H ^ v G
Epoch 5:
best score: 0.41
v < > >
> H > H
< ^ > H
H ^ v G
Epoch 6:
best score: 0.4
> < > >
> H > H
< ^ > H
H ^ v G
Epoch 7:
best score: 0.7
> v v ^
> H > H
< ^ > H
H v ^ G
Epoch 8:
best score: 0.63
> v ^ ^
> H > H
< ^ > H
H v v G
Epoch 9:
best score: 0.71
> v v ^
> H > H
< ^ > H
H v ^ G
Epoch 10:
best score: 0.7
> v v ^
> H > H
< ^ > H
H v ^ G
Epoch 11:
best score: 0.78
> < ^ <
> H > H
< ^ > H
H v ^ G
Epoch 12:
best score: 0.79
> < ^ <
> H > H
< ^ > H
H v ^ G
Epoch 13:
best score: 0.76
> < ^ <
> H v H
< ^ > H
H v ^ G
Epoch 14:
best score: 0.77
> < ^ <
> H v H
< ^ > H
H v ^ G
Epoch 15:
best score: 0.78
> < ^ <
> H v H
< ^ > H
H v ^ G
Epoch 16:
best score: 0.81
> < v <
> H > H
< ^ > H
H v ^ G
Epoch 17:
b[...]
{'fraction_non_alphanumeric': 0.1247791519434629, 'fraction_numerical': 0.029814487632508834, 'mean_word_length': 3.444062806673209, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 92, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '644894', 'license': 'MIT', 'n_tokens_mistral': 3542, 'n_tokens_neox': 3404, 'n_words': 1092, 'path': '0.frozenlake.ipynb', 'repo_name': 'Hulk89/Reinforcement_Learning', 'size': '9056'}