showing projection is a linear operator Show that the orthogonal projection is linear.
Let $x_i=y_i+z_i$, where $x_i\in X$, $y_i\in Y$, $z_i\in Y^\perp$, and $\alpha,\beta$ be scalars. Then \begin{align}P(\alpha x_1+\beta x_2)&=P(\alpha(y_1+z_1)+\beta(y_2+z_2))\\&=P(\alpha y_1+\beta y_2+\alpha z_1+\beta z_2)\end{align}
Now I know $\alpha x_1 + \beta y_1 \in Y$ but not sure what to do here.

The key here is that $X = Y\oplus Y^\perp$, i.e. for any $x\in X$ there are unique $y\in Y, z\in Y^\perp$ such that $x = y+z$. 
This is essential in order for $P(x) = P(y+z) = y$ to be well defined in the first place. 
Now, what you showed is that $\alpha x_1 +\beta x_2$ can be uniquely written as $(\alpha y_1 + \beta y_2) + (\alpha z_1 + \beta z_2)$ where $\alpha y_1 + \beta y_2\in Y$, $\alpha z_1 + \beta z_2\in Y^\perp$. So, what is $P(\alpha x_1 + \beta x_2)$?
Edit:
Your definition of orthogonal projection assumes that for each $x\in X$ there is unique $y\in Y$ such that $x-y\in Y^\perp$. This is actually equivalent to stating that $Y\oplus Y^\perp = X$, i.e. there are unique $y\in Y$, $z\in Y^\perp$ such that $x = y+z$ (notice that $z = x-y$ from your definition). 
So, assume that there is unique $y\in Y$ such that $x-y\in Y^\perp$. Then, $P(x) = P(y+(x-y)) = y$ is just restating my claim if you substitute $z = x-y$.
The bigger question is why such $y$ exists. In finite-dimensional case this follows immediately from existence of orthonormal basis for $Y$. Then you can define $y = \sum \langle x,e_i\rangle e_i$. In infinite-dimensional case we can use projection theorem for Hilbert spaces when we can find such $y \in Y$ that minimizes length $\|x-y\|$ (think of a point and a line: minimum distance between point and line is given by orthogonal projection).
I hope this clarifies things a bit.
