Q:

Getting nonphysical results when solving for the index of refraction of a slab?

I'm trying to computationally find the refractive indices (real and imaginary) for a thin slab suspended in air (so the only indices to deal with are air and my material's). I've experimentally taken transmission and reflection measurements of intensity in a certain wavelength range.
From a textbook chapter I read, 
$$r_{ij}[n_i, n_j]= (n_i - n_j)/(n_i + n_j)$$
$$t_{ij}[n_i, n_j]= (2 n_i)/(n_i + n_j)$$
$$r = (r_{ij}[n_1, n_2] + 
     r_{ij}[n_2, n_3] e^{2i \pi n_2 d/\lambda})/(1 + 
     r_{ij}[n_1, n_2] r_{ij}[n_2, n_3] e^{2i \pi n_2 d/\lambda})$$
$$t =  
  \sqrt{n3/n1} (t_{ij}[n_1, n_2] t_{ij}[n2, n3] e^{
       2i \pi n_2 d/\lambda})/(1 + 
      r_{ij}[n_1, n_2] r_{ij}[n_2, n_3] e^{2i \pi n_2 d/\lambda})$$
and $$T = |t|^2$$, $$R=|r|^2$$
for transmission and reflection of a plane wave through a 'slab' (though I may be missing a factor of 2 in a couple places. I've used Mathematica style brackets for the functions).
From here, $T$ and $R$ are both functions of $n_1$ (the air, ~1) and $n_2$ my material ($n_3$ is air again, here). So mathematically I should be able to figure out $n_2$ for each $T$ and $R$ pair, at each wavelength.
So I've done this. I have Mathematica (MM) find the $n_2$ that minimizes $$\sqrt{(T-T_{exp})^2+(R-R_{exp})^2}$$ at each wavelength. Then, to see how close it is, I plug $n_2$ back into the original equation and plot it against my experimental results to see how close they match up.
The problem is, they're matching up incredibly well -- but the results I'm getting for $n_2$ aren't realistic (more specifically, the real part of $n_2$ is negative, and this is not a negative refractive index material...). Here is a graphical example of what I mean (the horizontal axis is wavelength, in units of microns. The vertical axis is unitless for all):

And here I've plotted the values of $T$ and $R$ from my computed $n_2$'s (green and black) over the first graph. As you can see, they're so similar you can't tell, except at the leftmost tails.

What could be going on? One possibility is that though my solution for $n_2$ gives very close values for $T$ and $R$, there are very different values of $n_2$ that give even closer values. 
Someone I talked to told me that the plane wave model doesn't always apply at some scales -- which blew my mind because I have always seen it used. He said it was the far field solution to dipole radiation but that it might not apply at my length scale. Could anyone verify or refute this?

A:

Three possibilities I can think of:

A systematic shift in the data can certainly be relevant. T and R can be unphysical even if they're both positive and don't look unphysical. For example, I calculated T and R for a 40 nm slab of gold at 400-800 nm, then made fake experimental data where $R_{exp}=R$ and $T_{exp}=T+\Delta T$ and solved for n like you did. The solver started to give me negative $n_r$ for some wavelengths at $\Delta T=0.1$.
That model assumes that the light is incident at zero degrees and the material is not birefringent, and maybe those are not true in your experiment.
I checked the model using the transfer matrix formalism. It does seem like it might be off. The expression for r is correct (and that one I knew from memory anyway) but I get this for t, using shorthand $r_{12}$ for what you're calling $r_{ij}[n_1, n_2]$:
$$ t = \frac{t_{21}t_{32} e^{i\pi n_2 d/\lambda}} {1 + r_{12}r_{23} e^{2i\pi n_2 d/\lambda}} $$
(without the $\sqrt{n_3/n_1}$ which I prefer to put as $T=(n_3/n_1)|t|^2$.) That is, you've got an extra factor of 2 in the exponential in the numerator, and you have $t_{12}t_{23}$ instead of $t_{21}t_{32}$. Actually the latter shouldn't matter if $n_1=n_3$, but the former does. It's been too long for me to remember off the top of my head whether I did it correctly, but I can show my work if you disagree ;-)

