Q:

How can $a|0\rangle + b|1\rangle$ act on $x|00\rangle + y|11\rangle$?

How can the projective measurement $|A\rangle = a |0 \rangle + b |1 \rangle$ act on the two-qubit quantum state $|\Psi\rangle=x|00\rangle + y|11\rangle$?
edit:
Say A measures the first qubit of the two-qubit state. What's the probability of outcome 1?
See equation (6) here for more context.

A:

There's two steps to this

The first thing is to always act linearly with the bra:
\begin{align}
\langle A |\Psi\rangle 
& =
\langle A|\bigg[ x|00\rangle + y |11\rangle \bigg]
\\ & =
x\langle A|00\rangle + y \langle A|11\rangle
\end{align}
The second part is to understand how a single-party projector works on a bipartite tensor-product state. This requires either clearer notation or an explicit indication of what tensor factor the bra acts on; in this case, you've specified that it works on the first factor, so you have in general
$$
\langle A| uv\rangle 
= \langle A| \cdot \left(|u\rangle \otimes |v\rangle \right)
= \langle A|u\rangle \cdot |v\rangle,
$$
leaving you with a complex number, $\langle A|u\rangle$, multiplying a Bob-side state vector $|v\rangle$.

Putting the two together, you get the final result
\begin{align}
\langle A |\Psi\rangle 
& =
\langle A|\bigg[ x|00\rangle + y |11\rangle \bigg]
\\ & =
x\langle A|00\rangle + y \langle A|11\rangle
\\ & =
x\langle A|0\rangle |0\rangle + y \langle A|1\rangle |1\rangle
\\ & =
a^* x |0\rangle + b^* y |1\rangle,
\end{align}
i.e. a Bob-side state vector as above.

If you want to find out the probability of this outcome, you need to be slightly more specific, because you might be asking in correlation with a Bob-side measurement or not, and the correlated probabilities are different beasts, for starters because they depend on two independent variables instead of just one. 
If you want to ignore whatever happens on Bob's side, that's also just fine, but even then it still pays off to build a complete picture of the projective measurement you're building, which would normally be on the basis
\begin{align}
|A\rangle & = \phantom{+} a |0 \rangle + b |1 \rangle,\\
|A^\perp\rangle & = -b^* |0 \rangle + a^* |1 \rangle.
\end{align}
(Two things to keep in mind: (i) the two-outcome measurement along a basis is not the most general projective measurement you can do; by coupling to an ancilla system you can implement a POVM with an arbitrary number of outputs, and (ii) the state $|A_1(\theta)\rangle$ in the paper you referred to is not the orthogonal complement to $|A_0(\theta)\rangle$ and it is not the other outcome of the first measurement; instead, $|A_1(\theta)\rangle$ represents a completely different measurement Alice could do. And yes, that is confusing notation and frankly it's reprehensible that PRL let it through.)
Once you do that, then you have two possible outcomes, $|A\rangle$ and $|A^\perp \rangle$, and the probability for the former is
\begin{align}
\left\|\langle A |\Psi\rangle \right\|^2
& =
\left\|a^* x |0\rangle + b^* y |1\rangle\right\|^2
\\ & = |a^*x|^2 + |b^*y|^2.
\end{align}
It's then a good exercise to check that this adds with the probability of getting outcome $|A^\perp\rangle$ (while also ignoring Bob's side) to give unity.

