Q:

Find the probability of certain measurement for a Laplace Operator on a state function

Let $H$ be the operator $ -\frac{d^{2}}{dx^{2}} $ and let its domain be $$\{f\in L^{2}(\mathbb{R},d\lambda)\text{ }:\int_{-\infty}^{\infty}|xF[f(x)]|^{2}dx<\infty\} $$ where $F$ is the Fourier Transform.
Let $h(x)$ be the state vector $h(x)=\begin{cases}
\frac{1}{\sqrt{2}} & x\in[0,2]\\
0 & \text{else }\end{cases}$.

Find the probability that a measurement will return values in $[\frac{1}{2},1]$.
================
Here is my own response, which I am very skeptical of.

The General Idea

Find the eigenspace of of the opeartor $H$ spanned by its eigenvectors (eigfunctions) $\psi_{k}(x)$.
Express $h(x)$ as a linear combination of the eigenfunctions: $h(x)=\sum C_{k}\psi_{k}(x)$.
Based on this document, I believe that the if $\{C_{k}\}$ is the set of all possible eigenvalues, then the probability a measurement of the observable on $H$ on the state returns value in the set $$\Omega\subseteq\{C_{k}\}$$ is

$$\frac{\sum_{k\in\Omega}|C_{k}|^{2}}{\sum_{k\in\{C_{k}\}}|C_{k}|^{2}}$$
The Calculation
1.Eigenspace of $H$
Let $\lambda$ be the eigenvalue of $H$, then $$H[f]=\lambda f(x)\implies f''(x)=-\lambda f(x)$$
$$f(x)=c_{1}e^{i\sqrt{\lambda}x}+c_{2}e^{-i\sqrt{\lambda}x}$$
Thus the eigenvalues of $H$ are $\lambda\in(-\infty,+\infty)$
while correspoinding eigenfunctions for each eigenvalue $\lambda$ are $e^{i\sqrt{\lambda}x}$ and $e^{-i\sqrt{\lambda}x}$.
2.Decomposition of the state function
For our state $$h(x)=\begin{cases}
\frac{1}{\sqrt{2}} & x\in[0,2]\\
0 & \text{else}\end{cases}$$
we can decompose into the the space spanned by eigenfunctions of $H$ using Fourier Transform, which states that in general: 
$$f(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\hat{f(x)}e^{ikx}dx$$and $$\hat{f(x)}=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}f(x)e^{-ikx}dx$$
For our state, Fourier Transformation produces
$$h(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\hat{h(x)}e^{ikx}dk$$ where $$\hat{h(x)}=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}h(x)e^{-ikx}dx=\frac{1}{\sqrt{2\pi}}\int_{0}^{2}\frac{1}{\sqrt{2}}e^{-ikx}dx=\frac{1}{2\sqrt{\pi}}\frac{1}{-ik}e^{-ikx}|_{0}^{2}$$ $$=\frac{i}{2k\sqrt{\pi}}(e^{-2ik}-1)$$
Thus
$$h(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\frac{i}{2k\sqrt{\pi}}(e^{-2ik}-1)e^{ikx}dk$$
Expressing $h(x)$ as a linear combination of the basis of the eigenspace of $H$, which is $\{e^{ikx}\}$, we have $$h(x)=\sum_{k\in\mathbb{R}}C_{k}e^{ikx}$$
where $$C_{k}=\frac{i(e^{-2ik}-1)}{2\sqrt{2}\pi k}$$
3.The probability
To have a measure result in $[\frac{1}{2},1]$, $\lambda=[\frac{1}{2},1]$ means that the eigenfuctions are $$e^{\pm i\sqrt{\lambda}x}=e^{\pm ikx}$$ where $$k\in[-1,-\frac{1}{\sqrt{2}}]\cup[\frac{1}{\sqrt{2}},1]=\Omega$$.
The probability that this event occurs is 
$$\sum_{k\in\Omega}|C_{k}|^{2}/\sum_{k\in\mathbb{R}}|C_{k}|^{2}$$
Since the summation is over a continuous range of k, (this is where the the descrete idea from this document is extended in an continuous case, which I am not sure if it is correct), we can compute it using an integral:
$$\sum_{k\in\text{all}k}|C_{k}|^{2}=\int_{-\infty}^{\infty}\left|\frac{i(e^{-2ik}-1)}{2\sqrt{2}\pi k}\right|^{2}dk$$
$$=\int_{-\infty}^{\infty}\frac{e^{-2ik}-1}{2\sqrt{2}\pi k}\cdot\frac{e^{2ik}-1}{2\sqrt{2}\pi k}dk$$ $$=\int_{-\infty}^{\infty}\frac{1-e^{-2ik}-e^{2ik}+1}{8\pi^{2}k^{2}}dk$$
$$=\frac{1}{8\pi^{2}}\int_{-\infty}^{\infty}\frac{2-2\cos2k}{k^{2}}dk=$$ $$\frac{1}{4\pi^{2}}\int_{-\infty}^{\infty}\frac{1-\cos2k}{k^{2}}dk=0.159154 $$
and similarly $$\sum_{k\in\Omega}|C_{k}|^{2}=\int_{-1}^{-1/\sqrt{2}}\left|\frac{e^{-2ik}-1}{2\sqrt{2}\pi k}\right|^{2}dk+\int_{\frac{1}{\sqrt{2}}}^{1}\left|\frac{e^{-2ik}-1}{2\sqrt{2}\pi k}\right|^{2}dk$$ $$\frac{1}{2\pi^{2}}\int_{1/\sqrt{2}}^{1}\frac{1-\cos2k}{k^{2}}dk=0.00230997$$
Thus the probability is $$=0.00230996/0.159154=0.145$$ or 14.5%

A:

The calculations and result seem to be correct (except that you've got the integration bounds for $[-1,-1/\sqrt2]$ the wrong way around, and the end result is $0.145$, not $0.146$; the fraction leading to that value is OK, though).
But you're quite right to be skeptical about the way you generalized the discrete result to the continuous case. The continuous case is somewhat tricky in that the individual eigenfunctions in the continuous spectrum are not normalizable (try normalizing $\mathrm e^{\mathrm ikx}$). The formulas involving sums that you found for the discrete case assume that you have an orthonormal basis of discrete eigenstates; then the probability can be calculated simply as a weighted average of the eigenvalues with the squared amplitudes as weights. In the continuous case one would first need to define what it means for a continuous spectrum of eigenfunctions to form an orthonormal basis. There are various levels of rigour at which this can be handled. A common approach in physics is not to worry about this too much and to write completeness and orthonormality relations using delta "functions":
$$\int|k\rangle\langle k|\mathrm dk=1\;,$$
$$\int\langle k|k'\rangle\mathrm dk=\delta(k-k')\;.$$
This can be made more rigorous using the theory of distributions, where the delta "function" becomes a well-defined distribution. Another approach is to associate projection operators with subsets of the continuous spectrum. You didn't have to deal with any of this because things work out nicely for the Fourier transform, but I'd maintain that healthy skepticism with respect to replacing discrete sums by "sums over $\mathbb R$" and the like.
By the way, you can put the amplitude
$$C_{k}=\frac{\mathrm i(\mathrm e^{-2\mathrm ik}-1)}{2\sqrt{2}\pi k}$$
into a more recoginzable form by noting that the given wave function $h(x)$ is just a rectangular pulse shifted away from the origin, and that shift corresponds to a phase factor in Fourier space, and if you take out that phase factor you should get the Fourier transform of a rectangular pulse, which is the sinc function. And indeed:
$$
\begin{eqnarray}
C_{k}
&=&
\frac{\mathrm i(\mathrm e^{-2\mathrm ik}-1)}{2\sqrt{2}\pi k}\\
&=&
\mathrm e^{-\mathrm ik}\frac{\mathrm i(\mathrm e^{-\mathrm ik}-e^{\mathrm ik})}{2\sqrt{2}\pi k}\\
&=&
\mathrm e^{-\mathrm ik}
\frac1{\sqrt2\pi k}\frac{(\mathrm e^{\mathrm ik}-e^{-\mathrm ik})}{2\mathrm i}\\
&=&
\mathrm e^{-\mathrm ik}
\frac1{\sqrt2\pi}\frac{\sin k}k\\
&=&
\mathrm e^{-\mathrm ik}
\frac1{\sqrt2\pi}\operatorname{sinc}k\;.
\end{eqnarray}
$$

