<jupyter_start><jupyter_text>Conditional Expectation As a Projection  Suppose we are trying to predict the value of a random variable $Y$ based on a related random variable $X$. As you saw in Data 8, a natural method of prediction is to use the "center of the vertical strip" at the given value of $X$.Formally, given $X=x$, we are proposing to predict $Y$ by $E(Y \mid X=x)$.<jupyter_code># NO CODE
plot_curved_scatter()
xstar = 1.25
ystar = 3
tbl = Table().with_columns('x', z, 'y', y)
strip1 = tbl.where('x', are.between(xstar-0.01, xstar+0.012))
plt.plot(strip1.column(0), strip1.column(1), color='darkblue')
strip2 = tbl.where('x', are.between(-0.01, 0.012))
plt.plot(strip2.column(0), strip2.column(1), color='darkblue')
x = np.arange(xmin, xmax+0.01, 0.01)
plt.plot(x, x**2, color='gold', lw=3)
plt.title('Gold Curve $E(Y | X=x)$: Centers of Vertical Strips');<jupyter_output><empty_output><jupyter_text>The conditional expectation $E(Y \mid X)$ is the function of $X$ defined by$$b(x) ~ = ~ E(Y \mid X = x)$$We are using the letter $b$ to signifiy the "best guess" of $Y$ given the value of $X$. Later in this chapter we will make precise the sense in which it is the best. In random variable notation,$$E(Y \mid X) ~ = ~ b(X)$$For a point $(X, Y)$, the error in this guess is$$D_w = Y - b(X)$$The subscript $w$ reminds us that this error is a deviation *within* a vertical strip – it is the difference between $Y$ and the center of the strip at the given value of $X$.<jupyter_code># NO CODE
plot_curved_scatter()
x = np.arange(xmin, xmax+0.01, 0.01)
#plt.plot([xmin, xmax], [1, 1], color='k', lw=3)
plt.plot(x, x**2, color='gold', lw=3)
xstar = 1.25
ystar = 3
plt.plot([xstar, xstar], [ystar, xstar**2], color='purple', lw=3)
#plt.plot([xstar, xstar], [xstar**2, 1], color='lime', lw=3)
plt.scatter(xstar, ystar, color='darkblue', s=30, zorder=3)
plt.title('$D_w = $ Length of Purple Segment')
plt.xlim(-2, 2)
plt.ylim(-3, 6);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1688708036622584, 'fraction_numerical': 0.03255340793489318, 'mean_word_length': 5.089783281733746, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '308054', 'license': 'MIT', 'n_tokens_mistral': 798, 'n_tokens_neox': 744, 'n_words': 253, 'path': 'content/Chapter_22/01_Conditional_Expectation_Projection.ipynb', 'repo_name': 'dcroce/jupyter-book', 'size': '1966'}