Markov decision process Markov chain probability distribution instantaneous
Formally the environment is modeled as a Markov decision process (MDP) with states 






s

1


,
.
.
.
,

s

n



∈
S



{\displaystyle \textstyle {s_{1},...,s_{n}}\in S}

 and actions 






a

1


,
.
.
.
,

a

m



∈
A



{\displaystyle \textstyle {a_{1},...,a_{m}}\in A}

. Because the state transitions are not known, probability distributions are used instead: the instantaneous cost distribution 




P
(

c

t



|


s

t


)



{\displaystyle \textstyle P(c_{t}|s_{t})}

, the observation distribution 




P
(

x

t



|


s

t


)



{\displaystyle \textstyle P(x_{t}|s_{t})}

 and the transition distribution 




P
(

s

t
+
1



|


s

t


,

a

t


)



{\displaystyle \textstyle P(s_{t+1}|s_{t},a_{t})}

, while a policy is defined as the conditional distribution over actions given the observations. Taken together, the two define a Markov chain (MC). The aim is to discover the lowest-cost MC.
