How can I prove this expression not to be a characteristic function Let $\phi$ be a function of two real arguments defined as follows:
$$ \phi\left(t_1, t_2\right) = \exp \left(-t_1^2-t_2^2 +i \frac{t_1}{3}\frac{ t_1^2-3 t_2^2 }{t_1^2+t_2^2} \right)$$
and whenever $t_1^2 + t_2^2 >0$ and $\phi(0,0)$ is defined to be $1$.
How can I go about proving that this is not a characteristic function of a 2D random vector. This is function is known not to correspond to any 2D random measure, and is a special case of expression considered by D.J. Marcus in "Non-stable laws with all projections stable" (see first link in Google Scholar).
Notice that both $\phi(t_1, 0) = \exp( -t_1^2 + i t_1/3)$ and $\phi(0, t_2) = \exp(-t_2^2)$ are characteristic functions of normal variates.
What techniques can be used to prove that $\phi$ corresponds to no probability measure on a plane ?

Suppose that $\phi$ was the characteristic function of a 2d random vector $X=(X_1,X_2)$. Then, $X_1$ and $X_2$ have the characteristic functions $\phi(t,0)=\exp(-t^2+it/3)$ and $\phi(0,t)=\exp(-t^2)$ respectively. This means that $X_1$ and $X_2$ are each normally distributed (but not joint normal). Then,
$$
\begin{align}
\exp(t\Vert X\Vert)&\le\exp(t\vert X_1\vert)\exp(t\vert X_2\vert)\\
&\le\max\left(e^{t\vert X_1\vert},e^{t\vert X_2\vert}\right)^2\\
&\le e^{2t\vert X_1\vert}+e^{2t\vert X_2\vert}
\end{align}
$$ has finite mean for all positive $t$. This means that $\phi(t_1,t_2)=\mathbb{E}[\exp(it_1X_1+it_2X_2)]$ extends to an analytic function of $t_1,t_2$ everywhere on $\mathbb{C}^2$. By uniqueness of analytic continuation, this must agree with your expression on $\mathbb{C}^2$. However, your expression does not converge as $t_2\to it_1$.

I'll add another proof that $\phi$ is not a characteristic function, which is more closely related to the title of the paper referenced in the question (Non-stable laws with all projections stable). If $\phi$ was the characteristic function of a random vector $X=(X_1,X_2)$ then, for any $a\in\mathbb{R}^2$, $a\cdot X$ would have characteristic function
$$
\mathbb{E}\left[e^{ita\cdot X}\right]=\phi(ta_1,ta_2)=\exp\left(-\Vert a\Vert^2 t^2+it\frac{a_1}{3}\frac{a_1^2-3a_2^2}{a_1^2+a_2^2}\right).
$$
So, $a\cdot X$ is normally distributed with variance $2\Vert a\Vert^2$ and mean $\mu(a)\equiv\frac{a_1}{3}\frac{a_1^2-3a_2^2}{a_1^2+a_2^2}$. This is contradictory, as it does not even respect additivity of expectations,
$$
\mathbb{E}[(a+b)\cdot X]=\mu(a+b)\not=\mu(a)+\mu(b)=\mathbb{E}[a\cdot X]+\mathbb{E}[b\cdot X].
$$
