<commit_before>// \file mainfunction.txt
/*! 
  \page IShouldrun.py Getting Started: A 10-15 Minute Introduction
  \section First Run
  
Welcome to RL-Python!\n
If you receive errors during any of the steps below, please refer to install.txt for solutions to common issues. \n \n

Begin by opening IShouldRun.py.  You will notice a series of parameters at the top of the file, followed by assignments to each of the functional components shown in \ref big_pic "The Big Picture": domain, representation, policy, agent, and experiment.\n
Leave these alone for now; just run the file as-is.  You should see something like the following:

\image html pitMaze_learning.png "PitMaze Domain Visualization (IShouldRun.py)"

This is a visual representation of the domain (here, "PitMaze") and is useful in quickly judging or demonstrating the performance of an experiment.  The objective on this domain is to move the agent (triangle) from the start (blue) to the goal (green) location in the shortest distance possible, while avoiding the pits (red); -0.001 reward is applied for every step, and reaching the goal or pit regions give rewards of +1.0 and -1.0 respectively, terminating the episode. \n
On the left, you can see the learned policy in action after each 1,000-step batch of training data.  Notice how the policy gradually converges to the optimal, direct route which avoids pits.
On the right, you can see the representation of the value function overlayed on the domain.  Notice how after successive iterations, the agent learns the high (green) value of being in states that lie along the optimal path, even though they offer no immediate reward.  It also learns the low (red) value of unimportant / undesirable states.\n
The set of possible actions in each grid is highlighted by arrows, where the size of arrows correspond to the Q(s,a). The best action is shown as black. Each domain in RL-Python offers visualization like that shown on the left, and where possible, a representation of the value function like that shown on the right as well.

    \section Interpreting Output
In the console window you should see output similar to the following:
[[ Frame this somehow as code / output / make it stand out?]]
...
647: E[0:00:01]-R[0:00:15]: Return=+0.97, Steps=33, Features = 20
1000 >>> E[0:00:04]-R[0:00:37]: Return=+0.99, Steps=11, Features = 20
1810: E[0:00:05]-R[0:00:23]: Return=+0.98, Steps=19, Features = 20
...

The meaning of each component is shown in the figure.  Note that a "performance run" (indicated by ">>>" in the output window) tests the agent using its latest policy, without any exploration or modifications that might be used during learning (such as the randomization of the episilon-greedy policy).  The visualization shows these performance runs. \n
(Note: If you see *only* performance runs as console output, this simply means that your machine is completing the batches of training data faster than the console logging rate, usually 1 Hz, and does not indicate a problem.)

\image html rl-python_output.png "Sample Output (IShouldRun.py)"

After the cutoff of 2,000 steps specified in IShouldRun.py, the experiment is complete, and a second figure window appears, showing the reward earned on each performance run.  On this domain, a near-optimal policy is located almost immediately (reward 0.989).

    \section Analyzing Data
The variable PROJECT_PATH in IShouldRun.py determines the directory in which all results are stored; IShouldRun.py defaults to ./Results/IShouldRun.  A folder for a particular experiment is automatically generated in that directory based on its parameters; here, "PitMaze-Tabular-2000" for the PitMaze domain, Tabular representation, with 2,000 steps total. \n
The data logged to the console is stored in a file "#-out.txt", where "#" is the ID of the experiment, here "1".  Open this "1-out.txt" and verify this. \n
Data is stored in a more compact form in "#-results.txt", described below. \n
\n
Now open the file "IShouldMerge.py" and run it.  (If you get an error which includes "No directory including result was found at ['Results/IShouldRun']", make sure you allow IShouldRun.py to run to completion, when the second figure window appears, approximately 30 seconds). \n
You should now see several figures which demonstrate various performance measures of the experiment, such as return vs. number of steps.  The "paths" variable in this file specifies a directory to recursively search to generate these figures (and where to store them); if multiple results directories are found, they are plotted simultaneously against each other on the same figure.  If multiple results files are found in the same directory (e.g. 1-results.txt, 2-results.txt, etc., generated using multipleRuns.py) the mean curve is drawn with variance around it. \n
\n
    \section A Slightly More Challenging Domain: Inverted Pendulum
The small PitMaze domain is easy to understand, but does not produce meaningful final plots because the optimal policy is learned so quickly.  We will now run an experiment on the Inverted Pendulum Domain, where the goal is to prevent the pendulum from falling below the horizontal by applying clockwise (red) or counterclockwise (black) torque.  No reward is received for balancing and there is no control penalty; a penalty is only applied if the pendulum falls, after which the episode terminates.\n
Return to IShouldRun.py, and:
1) Near the top (line 33), change the following two variables to allow more steps for learning the domain: \n
    PERFORMANCE_CHECKS  = 5 \n
    LEARNING_STEPS      = 8000 \n
\n
2) Scroll to the "Domain" section (line 62), comment the line corresponding to PitMaze and decomment the line corresponding to Pendulum_InvertedBalance. \n
Now, run IShouldRun.py and observe the new domain.  Initially, the agent fails to balance the pendulum, but after approximately 6000 steps, can do so reliably up to the maximum number of steps allowed for an episode, 300. \n
The value function, which plots pendulum angular rate against its angle, demonstrates the highly undesirable states of a steeply inclined pendulum (near the horizontal) with high angular velocity in the direction in which it is falling. \n
The policy initially appears random, but converges to the shape shown below, with distinct black (counterclockwise torque action) and red (clockwise action) regions in the first and third quadrants respectively, and a white stripe along the major diagonal between.  This makes intuitive sense; if the pendulum is left of center and/or moving counterclockwise (third quadrant), for example, a corrective clockwise torque action should certainly be applied.  The white stripe in between shows that no torque should be applied to a balanced pendulum with no angular velocity, or if it lies off-center but has angular velocity towards the balance point. \n

[[need figure demonstrating the above example with torques?]]

\n
To allow the experiment to run more quickly, without visualization: \n
1) Assign SHOW_PERFORMANCE    = 0 (surpresses visualization) \n
2) Assign PERFORMANCE_CHECKS  = 10 \n
3) Assign LEARNING_STEPS      = 20000 \n
4) Assign domain = Pendulum_InvertedBalance(episodeCap = 3000, logger = logger); \n
(Step 4 extends the maximum number of steps per episode from 300 to 3000 to make the task slightly more challenging.)
\n
Finally, return to IShouldMerge.py, comment the old "paths" variable, and decomment the one corresponding to the Pendulum results.  The learning up to approximately 5,000 steps should be apparent through continuously improving performance.  Large fluctuations in reward due to chance are smoothed by running the experiment many times using multipleRuns.py, described in another tutorial; for the experiment just performed, 30 runs yields \n

\section Conclusion
We have seen how to run experiments, interpret visualization, and generate results on two classic Reinforcement Learning domains.  We have also seen the structure of the high-level files;\n
IShouldRun.py <--> main.py - Identical, former has excess options removed.  You may now try using main.py instead of IShouldRun.py, experimenting with different agents, representations, etc., all of which have parameters defined at the top of main.py. \n
IShouldMerge.py <--> mergeRuns.py - Identical, former just has "paths" prespecified.  You should now use mergeRuns.py to generate your results.
\n
Finally, you should explore the rest of the structure of the framework, and perhaps try implementing domains and algorithms of your own. \n
\n

If you have questions, email <NAME> (agf[at]csail[dot]mit[dot]edu) or <NAME> (rhklein[at]mit[dot]edu).
\n
\n
"The only real mistake is the one from which we learn nothing." -<NAME>

*/
<commit_msg>fixed title issue<commit_after>// \file mainfunction.txt
/*! 
  \page IShouldrun.py Getting Started: A 10-15 Minute Introduction
  \section First_Run First Run
  
Welcome to RL-Python!\n
If you receive errors during any of the steps below, please refer to install.txt for solutions to common issues. \n \n

Begin by opening IShouldRun.py.  You will notice a series of parameters at the top of the file, followed by assignments to each of the functional components shown in \ref big_pic "The Big Picture": domain, representation, policy, agent, and experiment.\n
Leave these alone for now; just run the file as-is.  You should see something like the following:

\image html pitMaze_learning.png "PitMaze Domain Visualization (IShouldRun.py)"

This is a visual representation of the domain (here, "PitMaze") and is useful in quickly judging or demonstrating the performance of an experiment.  The objective on this domain is to move the agent (triangle) from the start (blue) to the goal (green) location in the shortest distance possible, while avoiding the pits (red); -0.001 reward is applied for every step, and reaching the goal or pit regions give rewards of +1.0 and -1.0 respectively, terminating the episode. \n
On the left, you can see the learned policy in action after each 1,000-step batch of training data.  Notice how the policy gradually converges to the optimal, direct route which avoids pits.
On the right, you can see the representation of the value function overlayed on the domain.  Notice how after successive iterations, the agent learns the high (green) value of being in states that lie along the optimal path, even though they offer no immediate reward.  It also learns the low (red) value of unimportant / undesirable states.\n
The set of possible actions in each grid is highlighted by arrows, where the size of arrows correspond to the Q(s,a). The best action is shown as black. Each domain in RL-Python offers visualization like that shown on the left, and where possible, a representation of the value function like that shown on the right as well.

    \section Interpreting_Output Interpreting Output
In the console window you should see output similar to the following:
[[ Frame this somehow as code / output / make it stand out?]]
...
647: E[0:00:01]-R[0:00:15]: Return=+0.97, Steps=33, Features = 20
1000 >>> E[0:00:04]-R[0:00:37]: Return=+0.99, Steps=11, Features = 20
1810: E[0:00:05]-R[0:00:23]: Return=+0.98, Steps=19, Features = 20
...

The meaning of each component is shown in the figure.  Note that a "performance run" (indicated by ">>>" in the output window) tests the agent using its latest policy, without any exploration or modifications that might be used during learning (such as the randomization of the episilon-greedy policy).  The visualization shows these performance runs. \n
(Note: If you see *only* performance runs as console output, this simply means that your machine is completing the batches of training data faster than the console logging rate, usually 1 Hz, and does not indicate a problem.)

\image html rl-python_output.png "Sample Output (IShouldRun.py)"

After the cutoff of 2,000 steps specified in IShouldRun.py, the experiment is complete, and a second figure window appears, showing the reward earned on each performance run.  On this domain, a near-optimal policy is located almost immediately (reward 0.989).

    \section Analyzing_Data Analyzing Data
The variable PROJECT_PATH in IShouldRun.py determines the directory in which all results are stored; IShouldRun.py defaults to ./Results/IShouldRun.  A folder for a particular experiment is automatically generated in that directory based on its parameters; here, "PitMaze-Tabular-2000" for the PitMaze domain, Tabular representation, with 2,000 steps total. \n
The data logged to the console is stored in a file "#-out.txt", where "#" is the ID of the experiment, here "1".  Open this "1-out.txt" and verify this. \n
Data is stored in a more compact form in "#-results.txt", described below. \n
\n
Now open the file "IShouldMerge.py" and run it.  (If you get an error which includes "No directory including result was found at ['Results/IShouldRun']", make sure you allow IShouldRun.py to run to completion, when the second figure window appears, approximately 30 seconds). \n
You should now see several figures which demonstrate various performance measures of the experiment, such as return vs. number of steps.  The "paths" variable in this file specifies a directory to recursively search to generate these figures (and where to store them); if multiple results directories are found, they are plotted simultaneously against each other on the same figure.  If multiple results files are found in the same directory (e.g. 1-results.txt, 2-results.txt, etc., generated using multipleRuns.py) the mean curve is drawn with variance around it. \n
\n
    \section Inv_Pend A Slightly More Challenging Domain: Inverted Pendulum
The small PitMaze domain is easy to understand, but does not produce meaningful final plots because the optimal policy is learned so quickly.  We will now run an experiment on the Inverted Pendulum Domain, where the goal is to prevent the pendulum from falling below the horizontal by applying clockwise (red) or counterclockwise (black) torque.  No reward is received for balancing and there is no control penalty; a penalty is only applied if the pendulum falls, after which the episode terminates.\n
Return to IShouldRun.py, and:
1) Near the top (line 33), change the following two variables to allow more steps for learning the domain: \n
    PERFORMANCE_CHECKS  = 5 \n
    LEARNING_STEPS      = 8000 \n
\n
2) Scroll to the "Domain" section (line 62), comment the line corresponding to PitMaze and decomment the line corresponding to Pendulum_InvertedBalance. \n
Now, run IShouldRun.py and observe the new domain.  Initially, the agent fails to balance the pendulum, but after approximately 6000 steps, can do so reliably up to the maximum number of steps allowed for an episode, 300. \n
The value function, which plots pendulum angular rate against its angle, demonstrates the highly undesirable states of a steeply inclined pendulum (near the horizontal) with high angular velocity in the direction in which it is falling. \n
The policy initially appears random, but converges to the shape shown below, with distinct black (counterclockwise torque action) and red (clockwise action) regions in the first and third quadrants respectively, and a white stripe along the major diagonal between.  This makes intuitive sense; if the pendulum is left of center and/or moving counterclockwise (third quadrant), for example, a corrective clockwise torque action should certainly be applied.  The white stripe in between shows that no torque should be applied to a balanced pendulum with no angular velocity, or if it lies off-center but has angular velocity towards the balance point. \n

[[need figure demonstrating the above example with torques?]]

\n
To allow the experiment to run more quickly, without visualization: \n
1) Assign SHOW_PERFORMANCE    = 0 (surpresses visualization) \n
2) Assign PERFORMANCE_CHECKS  = 10 \n
3) Assign LEARNING_STEPS      = 20000 \n
4) Assign domain = Pendulum_InvertedBalance(episodeCap = 3000, logger = logger); \n
(Step 4 extends the maximum number of steps per episode from 300 to 3000 to make the task slightly more challenging.)
\n
Finally, return to IShouldMerge.py, comment the old "paths" variable, and decomment the one corresponding to the Pendulum results.  The learning up to approximately 5,000 steps should be apparent through continuously improving performance.  Large fluctuations in reward due to chance are smoothed by running the experiment many times using multipleRuns.py, described in another tutorial. \n

\section Conclusion Conclusion
We have seen how to run experiments, interpret visualization, and generate results on two classic Reinforcement Learning domains.  We have also seen the structure of the high-level files;\n
IShouldRun.py <--> main.py - Identical, former has excess options removed.  You may now try using main.py instead of IShouldRun.py, experimenting with different agents, representations, etc., all of which have parameters defined at the top of main.py. \n
IShouldMerge.py <--> mergeRuns.py - Identical, former just has "paths" prespecified.  You should now use mergeRuns.py to generate your results.
\n
Finally, you should explore the rest of the structure of the framework, and perhaps try implementing domains and algorithms of your own. \n
\n

If you have questions, email <NAME> (agf[at]csail[dot]mit[dot]edu) or <NAME> (rhklein[at]mit[dot]edu).
\n
\n
<EM> The only real mistake is the one from which we learn nothing. </EM> -<NAME>

*/

{'fraction_non_alphanumeric': 0.057393884071200364, 'fraction_numerical': 0.016887266088544045, 'mean_word_length': 4.4778125, 'pattern_counts': {'":': 2, '<': 15, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5692124', 'n_tokens_mistral': 4694, 'n_tokens_neox': 4329, 'n_words': 2607}