The expected payoff of a dice game There's a question in my Olympiad questions book which I can't seem to solve:


  You have the option to throw a die up to three times.  You will earn
  the face value of the die.  You have the option to stop after each
  throw and walk away with the money earned. The earnings are not additive. What is the expected payoff of this game?


I found a solution here but I don't understand it.
Let's suppose we have only 1 roll. What is the expected payoff? Each roll is equally likely, so it will show $1,2,3,4,5,6$ with equal probability. Thus their average of $3.5$ is the expected payoff.

Now let's suppose we have 2 rolls. If on the first roll, I roll a $6$, I would not continue. The next throw would only maintain my winnings of $6$ (with $1/6$ chance) or make me lose. Similarly, if I threw a $5$ or a $4$ on the first roll, I would not continue, because my expected payoff on the last throw would be a $3.5$. However, if I threw a $1,2$ of $3$, I would take that second round. This is again because I expect to win $3.5$.

So in the 2 roll game, if I roll a $4,5,6$, I keep those rolls, but if I throw a $1,2,3$, I reroll. Thus I have a $1/2$ chance of keeping a $4,5,6$, or a $1/2$ chance of rerolling. Rerolling has an expected return of $3.5$. As the $4,5,6$ are equally likely, rolling a $4,5$ or $6$ has expected return $5$. Thus my expected payout on 2 rolls is $.5(5) + .5(3.5) = 4.25$.

Now we go to the 3 roll game. If I roll a $5$ or $6$, I keep my roll. But now, even a $4$ is undesirable, because by rerolling, I'd be playing the 2 roll game, which has expected payout of $4.25$. So now the expected payout is $\frac{1}{3}(5.5) + \frac{2}{3}(4.25) = 4.\overline{66}$.

Does that make sense?
Can someone explain to me why the expectation value of the outcome is not the same as the markov solution?

Imagine you were playing such that you cash out after 500 tries (instead of 2 or 3 treis).  On your first roll, you roll a 4.  Would you stop at the first roll because it's higher than 3.5?  I think not.  Your odds of getting a six are almost certain.  

What is the expectation value of rolling a die twice, while only keeping the highest value?  We can compute this as follows:

How many ways can I roll a 1?  The answer is 1.  1 on my first roll. (1) 

How many ways can I roll a 2?  The answer is 2.  Either on the first roll, (2) or on the second roll, assuming my first roll was a 1 (1,2).

How many ways can I roll a 3?  The answer is 3.  Either on the first roll (3) or on one of the second rolls, assuming my first roll was a 1,2 or 3 (1,3) (2,3).

And so on...

Therefore, there is only 1 outcome where I'd make \$1, two outcomes where I'd make \$2, 3 outcomes where I'd make \$3 dollars, up to 6 outcomes where I'd make \$6.

The total number of outcomes, then, is (1+2+3+4+5+6) = 21.

The expectation value is then:

$&lt;E&gt; = \$1 \frac{1}{21} + \$2\frac{2}{21} + \$3\frac{3}{21}$

$&lt;E&gt; = \frac{1}{21} \sum_{n=1}^{6} n^2$

$&lt;E&gt; = \$91/21 $

$&lt;E&gt; = \$4.3\bar{3}$

I guess I'm trying to answer the question "how much would you pay to play this game"?  I'd pay up to $4.333.

The other solutions look more sophisticated, so maybe I'm blowing smoke.