I solve such a problem. Lets have a function $Y=\sum_{k=-\infty}^\infty i\hat Y e^{ik\pi y}$and then I have a function which is defined as $X=\sum_{k=-\infty}^\infty ik^2\hat Y e^{ik\pi y}$. I know the $Y$. The $i$ is imaginary unit. How can I compute the $X$? I think I do the FFT on $Y$ and obtain thus the $\hat Y$, right? And then I think I will do the backward FFT of function $f=ik^2\hat Y$. But what have I do with the summation index $k$ here in the $f$?It is right that $FFT(ik^2\hat Y)=X$ ?I'm not sure absolutely what to do with $k$ when the FFT sum is summated per $k$.
A function as a sum of serie of modified FFT coeff. of another function - multiplied by sum number
fourier analysis;fftw
It appears you are very close to your answer, so I am going to detail the process as a whole to clarify a couple small details.  If this doesn't answer your question, please comment and I will modify my answer.You know $Y$, so you evaluate it at a number of points over your interval and plug it into $FFT$.  This produces your $\hat Y$, which is a vector.  Let's denote each element in this vector as $\hat Y_k$.  You then have that $\hat X_k = i k^2 \hat Y_k$.  $\hat X$ is a vector the same length as $\hat Y$.  You then plug $\hat X_k$ back into your $FFT^{-1}$ to produce your X.  It appears the only thing you were missing was representing $\hat Y$ as a vector.  Note that the $i k^2$ can be any function of $k$ you want.