<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Markov decision process\n",
    "\n",
    "This week's methods are all built to solve __M__arkov __D__ecision __P__rocesses. In the broadest sense, an MDP is defined by how it changes states and how rewards are computed.\n",
    "\n",
    "State transition is defined by $P(s' |s,a)$ - how likely are you to end at state $s'$ if you take action $a$ from state $s$. Now there's more than one way to define rewards, but we'll use $r(s,a,s')$ function for convenience.\n",
    "\n",
    "_This notebook is inspired by the awesome_ [CS294](https://github.com/berkeleydeeprlcourse/homework/blob/36a0b58261acde756abd55306fbe63df226bf62b/hw2/HW2.ipynb) _by Berkeley_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For starters, let's define a simple MDP from this picture:\n",
    "<img src='https://upload.wikimedia.org/wikipedia/commons/thumb/a/ad/Markov_Decision_Process.svg/800px-Markov_Decision_Process.svg.png' width=300px>\n",
    "_img by MistWiz (Own work) [Public domain], via Wikimedia Commons_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-02T13:44:32.642838Z",
     "start_time": "2018-04-02T13:44:32.545142Z"
    }
   },
   "outputs": [],
   "source": [
    "# If you Colab, uncomment this please\n",
    "# !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/spring19/week02_value_based/mdp.py\n",
    "\n",
    "transition_probs = {\n",
    "    's0': {\n",
    "        'a0': {'s0': 0.5, 's2': 0.5},\n",
    "        'a1': {'s2': 1}\n",
    "    },\n",
    "    's1': {\n",
    "        'a0': {'s0': 0.7, 's1': 0.1, 's2': 0.2},\n",
    "        'a1': {'s1': 0.95, 's2': 0.05}\n",
    "    },\n",
    "    's2': {\n",
    "        'a0': {'s0': 0.4, 's1': 0.6},\n",
    "        'a1': {'s0': 0.3, 's1': 0.3, 's2': 0.4}\n",
    "    }\n",
    "}\n",
    "rewards = {\n",
    "    's1': {'a0': {'s0': +5}},\n",
    "    's2': {'a1': {'s0': -1}}\n",
    "}\n",
    "\n",
    "from mdp import MDP\n",
    "mdp = MDP(transition_probs, rewards, initial_state='s0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use MDP just as any other gym environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-02T13:44:34.203384Z",
     "start_time": "2018-04-02T13:44:34.199297Z"<commit_msg>Fix MDP definition and change illustration to SVG<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Markov decision process\n",
    "\n",
    "This week's methods are all built to solve __M__arkov __D__ecision __P__rocesses. In the broadest sense, an MDP is defined by how it changes states and how rewards are computed.\n",
    "\n",
    "State transition is defined by $P(s' |s,a)$ - how likely are you to end at state $s'$ if you take action $a$ from state $s$. Now there's more than one way to define rewards, but we'll use $r(s,a,s')$ function for convenience.\n",
    "\n",
    "_This notebook is inspired by the awesome_ [CS294](https://github.com/berkeleydeeprlcourse/homework/blob/36a0b58261acde756abd55306fbe63df226bf62b/hw2/HW2.ipynb) _by Berkeley_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For starters, let's define a simple MDP from this picture:\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/a/ad/Markov_Decision_Process.svg\" width=\"400px\" alt=\"Diagram by Waldoalvarez via Wikimedia Commons, CC BY-SA 4.0\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-02T13:44:32.642838Z",
     "start_time": "2018-04-02T13:44:32.545142Z"
    }
   },
   "outputs": [],
   "source": [
    "# If you Colab, uncomment this please\n",
    "# !wget -q https://raw.githubusercontent.com/yandexdataschool/Practical_RL/spring19/week02_value_based/mdp.py\n",
    "\n",
    "transition_probs = {\n",
    "    's0': {\n",
    "        'a0': {'s0': 0.5, 's2': 0.5},\n",
    "        'a1': {'s2': 1}\n",
    "    },\n",
    "    's1': {\n",
    "        'a0': {'s0': 0.7, 's1': 0.1, 's2': 0.2},\n",
    "        'a1': {'s1': 0.95, 's2': 0.05}\n",
    "    },\n",
    "    's2': {\n",
    "        'a0': {'s0': 0.4, 's2': 0.6},\n",
    "        'a1': {'s0': 0.3, 's1': 0.3, 's2': 0.4}\n",
    "    }\n",
    "}\n",
    "rewards = {\n",
    "    's1': {'a0': {'s0': +5}},\n",
    "    's2': {'a1': {'s0': -1}}\n",
    "}\n",
    "\n",
    "from mdp import MDP\n",
    "mdp = MDP(transition_probs, rewards, initial_state='s0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use MDP just as any other gym environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-02T13:44:34.203384Z",
     "start_time": "2018-04-02T13:44:34.199297Z"
{'fraction_non_alphanumeric': 0.21346116219917824, 'fraction_numerical': 0.06945803169634122, 'mean_word_length': 2.6358463726884778, 'pattern_counts': {'":': 48, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5813676', 'n_tokens_mistral': 2261, 'n_tokens_neox': 2046, 'n_words': 464}