<jupyter_start><jupyter_text>Assignment 1  Problem 2(a)  <NAME>  McGill ID: 260800927  COMP 767- Reinforcement Learning Winter 2019  Grid World Environment Original Grid world environment was taken from link: https://github.com/dennybritz/reinforcement-learning \I modified it with: (1) "Terminal states" at being     -Upper left (T1) with reward 1.0    - Upper right (T2) with reward 10.0    - Everywhere else reward 0.0(2) Inserted stochastic action condition:    - where it takes input "prob_p = P " probability of taking policy action    - and with probability (1-P) takes random actionEnvironment Directory : gridworld.py 			"""            				T1  o  o  T2                 o  o  o  o				 o  o  o  o				 x  o  o  o						"""  ReportI have written my findings from the experments after each section.   For grid size 5x5<jupyter_code>#############################
#
# Modified Policy Iteration
#
##############################
"""
grid_size = Input Modifies grid shape "default = 5"
prob_p = probability of choosing stochastic action "default = 0.9"
stop_itr = finite number of backups to the value function before an improvement step "defalt=10" 
Final_Result = if True; gives final Policy and Value function "default = False"
"""
!python Modified_Policy_Iteration.py --grid_size=5 --prob_p=0.9 --stop_itr=10 --Final_Result=True
!python Modified_Policy_Iteration.py --grid_size=5 --prob_p=0.7 --stop_itr=10 --Final_Result=True<jupyter_output>True
Epoch 0
Running Modified Policy Iteration; Policy Evaluation is completed
Value Function:
[[6.860e+00 2.000e-02 2.170e+00 1.203e+01 6.862e+01]
 [1.000e-02 7.000e-02 4.600e-01 2.230e+00 1.361e+01]
 [1.000e-02 3.000e-02 1.300e-01 8.200e-01 2.840e+00]
 [0.000e+00 1.000e-02 4.000e-02 1.500e-01 5.500e-01]
 [0.000e+00 0.000e+00 0.000e+00 2.000e-02 1.300e-01]]

Policy has not converged, updating new policy
---------------------------------------------
Epoch 1
Running Modified Policy Iteration; Policy Evaluation is completed
Value Function:
[[ 6.86  5.18 24.31 41.03 68.62]
 [ 5.18  8.97 15.32 25.85 43.23]
 [ 3.26  5.65  9.65 16.29 27.23]
 [ 2.05  3.56  6.08 10.26 17.16]
 [ 1.29  2.24  0.55  6.46 10.81]]

Policy has not converged, updating new policy
---------------------------------------------
Epoch 2
Running Modified Policy Iteration; Policy Evaluation is completed
Value Function:
[[ 6.86  5.18 24.31 41.03 68.62]
 [ 5.18  8.97 15.32 25.85 43.23]
 [ 3.26  5.65[...]<jupyter_text>Report: Even though the Policy isn't optimal but it finds it's way to terminal state as value function around Terminal states are higher. In this case when we choose stochastic action probability 0.7, gives higher value function around the upper right terminal comparing other states. For p = 0.7 and 0.9 policy tries to end up at upper right terminal<jupyter_code>#####################
#
# Policy Iteration
#
#####################
!python Policy_Iteration.py --grid_size=5 --prob_p=0.9 --Final_Result=True
!python Policy_Iteration.py --grid_size=5 --prob_p=0.7 --Final_Result=True<jupyter_output>Epoch 0
Running Policy Iteration; Value function Converged after 73 iterations
Value Function:
[[1.000e+01 1.640e+00 3.500e-01 1.500e-01 9.996e+01]
 [1.000e-02 4.000e-02 7.000e-02 4.200e-01 2.400e+00]
 [0.000e+00 1.000e-02 4.000e-02 1.500e-01 4.800e-01]
 [0.000e+00 0.000e+00 0.000e+00 2.000e-02 9.000e-02]
 [0.000e+00 0.000e+00 0.000e+00 0.000e+00 2.000e-02]]

Policy has not converged, updating new policy
---------------------------------------------
Epoch 1
Running Policy Iteration; Value function Converged after 73 iterations
Value Function:
[[1.000e+01 6.300e+00 3.970e+00 3.570e+00 9.996e+01]
 [3.600e-01 5.700e-01 2.500e+00 5.670e+00 9.000e+00]
 [2.200e-01 3.600e-01 2.250e+00 3.570e+00 5.670e+00]
 [1.400e-01 3.500e-01 1.420e+00 2.250e+00 3.570e+00]
 [3.500e-01 5.600e-01 8.900e-01 8.000e-02 2.250e+00]]

Policy has not converged, updating new policy
---------------------------------------------
Epoch 2
Running Policy Iteration; Value function Converged after 73 iterations
Value Functio[...]<jupyter_text>Report: Comparing with other algorithms Policy Iteration gives the optimal policy when probability of taking stochastic action is 0.9. For p = 0.7 for some reason policy indicates to terminate at upper left terminal even though upper right terminal state has higher value<jupyter_code>#########################
#
# Value Iteration
#
##########################
!python Value_Iteration.py --grid_size=5 --prob_p=0.9 --Final_Result=True
!python Value_Iteration.py --grid_size=5 --prob_p=0.7 --Final_Result=True<jupyter_output>Value Function after 1 iteration
[[ 1.    0.    0.    0.   10.  ]
 [ 0.63  0.06  0.    0.    6.3 ]
 [ 0.    0.04  0.02  0.01  3.97]
 [ 0.    0.02  0.01  0.01  2.5 ]
 [ 0.    0.01  0.01  0.01  1.58]]

Value Function after 21 iteration
[[ 8.91  8.27 13.39 21.63 89.06]
 [ 5.61  5.21 21.63 34.86 56.11]
 [ 5.21  8.43 13.62 21.96 35.35]
 [ 3.28  5.31  8.58 13.84 22.27]
 [ 2.07  3.35  5.41  8.72 14.03]]

Value Function after 41 iteration
[[ 9.87  9.72 15.47 24.59 98.67]
 [ 6.22  6.13 24.59 39.1  62.16]
 [ 6.13  9.74 15.49 24.64 39.16]
 [ 3.86  6.14  9.76 15.52 24.67]
 [ 2.43  3.87  6.15  9.78 15.54]]

Value Function after 61 iteration
[[ 9.98  9.9  15.72 24.95 99.84]
 [ 6.29  6.24 24.95 39.62 62.9 ]
 [ 6.24  9.9  15.72 24.96 39.63]
 [ 3.93  6.24  9.9  15.72 24.96]
 [ 2.48  3.93  6.24  9.91 15.73]]

Value Function after 81 iteration
[[10.    9.92 15.75 25.   99.98]
 [ 6.3   6.25 25.   39.68 62.99]
 [ 6.25  9.92 15.75 25.   39.68]
 [ 3.94  6.25  9.92 15.75 25.  ]
 [ 2.48  3.94  6.25  9.92 15.75[...]<jupyter_text>Report when P=0.7 both the terminal seems equally likely form value function where for P=0.9 upper right terminal state get's higher value function and makes the policy more likely to terminate with higher reward  Grid size 50X50<jupyter_code>###########################

# Modified Policy Iteration

###########################
!python Modified_Policy_Iteration.py --grid_size=50 --prob_p=0.9 --stop_itr=10 --Final_Result=True
!python Modified_Policy_Iteration.py --grid_size=50 --prob_p=0.7 --stop_itr=10 --Final_Result=True
###################

# Policy Iteration

###################
!python Policy_Iteration.py --grid_size=50 --prob_p=0.9 --Final_Result=True
!python Policy_Iteration.py --grid_size=50 --prob_p=0.7 --Final_Result=True
###################
#
# Value Iteration
#
####################
!python Value_Iteration.py --grid_size=50 --prob_p=0.9 --Final_Result=True
!python Value_Iteration.py --grid_size=50 --prob_p=0.7 --Final_Result=True<jupyter_output>Value Function after 1 iteration
[[ 1.    0.63  0.   ...  0.    0.   10.  ]
 [ 0.63  0.4   0.25 ...  0.    0.    0.  ]
 [ 0.4   0.25  0.16 ...  0.    0.    0.  ]
 ...
 [ 0.    0.    0.   ...  0.    0.    0.  ]
 [ 0.    0.    0.   ...  0.    0.    0.  ]
 [ 0.    0.    0.   ...  0.    0.    0.  ]]

Value Function after 21 iteration
[[ 8.91  5.61  1.38 ...  0.    0.   89.06]
 [ 5.61  3.53  2.23 ...  0.    0.    0.  ]
 [ 3.53  2.23  1.4  ...  0.    0.    0.  ]
 ...
 [ 0.    0.    0.   ...  0.    0.    0.  ]
 [ 0.    0.    0.   ...  0.    0.    0.  ]
 [ 0.    0.    0.   ...  0.    0.    0.  ]]

Value Function after 41 iteration
[[ 9.87  6.22  1.55 ...  0.    0.   98.67]
 [ 6.22  3.92  2.47 ...  0.    0.    0.  ]
 [ 3.92  2.47  1.55 ...  0.    0.    0.  ]
 ...
 [ 0.    0.    0.   ...  0.    0.    0.  ]
 [ 0.    0.    0.   ...  0.    0.    0.  ]
 [ 0.    0.    0.   ...  0.    0.    0.  ]]

Value Function after 61 iteration
[[ 9.98  6.29  1.57 ...  0.    0.   99.84]
 [ 6.29  3.96  2.5  ...  0.[...]
{'fraction_non_alphanumeric': 0.1902818014221754, 'fraction_numerical': 0.16934421912035819, 'mean_word_length': 2.2779456193353473, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '17764', 'license': 'MIT', 'n_tokens_mistral': 3679, 'n_tokens_neox': 2926, 'n_words': 859, 'path': 'Assignment01/Problem02/Assignment1_Problem2a.ipynb', 'repo_name': 'SaminYeasar/COMP-767-Reinforcement-Learniing', 'size': '7606'}