The unsolved mathematical light beam problem I have the following problem:
Imagine that you have a sphere sitting at the interface of two media(like water and oil). And the position(the heigth) of the interface to the center of the sphere is fixed and known(I drew a picture for two different situations). Now think about this: A beam of light (parallel rays) that enclose an angle alpha to the interface enters (the direction is supposed to be: they first enter the lower medium and then go to the upper one) hits this interface with the sphere. Now the question is: Can we find an analytical expression for the maximum area of the sphere perpendicular to the direction of the rays that is defined by the rays that enter the sphere without going first into the other medium? 
If this is unclear look at the picture: In situation 1 I drew 2 rays (of the infinitely many that enter at this angle) that fulfill the condition that they first hit the sphere without going in the upper medium. Now especially the left one is crucial since if I had chosen one that was even more slightly shifted to the left side, this one would have been in the upper medium, so no longer a reasonable candidate. The right one is not a restriction. The right picture is poorly drawn, since I wanted to have one, where both rays restrict the accessible area. But I think you have the idea now, but what do I mean by area?
I am looking for the biggest area(=PROJECTION OF THE SURFACE AREA of points that fulfill this on a plane)
 perpendicular to the direction of the rays inside the sphere, that consists of rays that enter the sphere and fulfill the property above . So in the first picture this would probably the area going through the center and "somewhat enclosed by the two arrays" and in the second part, this one should be the "imaginary interface" inside the sphere. 

If you have any questions, please do not hesitate to post them

Let us assume that the radius of the sphere is $R$, the height of the center above the separation plane is $h$ (if the center is below the plane, $h$ is negative, and the angle the rays make with the vertical line is $\beta\in(0,\pi/2)$ (it is $\beta=\frac \pi 2-\alpha$ on your picture).
Only the case $|h|<R$ is interesting. Now, 3 possibilities may present itself:
1) $h\le -R\sin\beta$. Then we want the area of the projection of the circle of radius $\sqrt{R^2-h^2}$ to the slanted plane. It is just $A=\pi(R^2-h^2)\cos\beta$.
2) $h> R\sin\beta$. Then the answer is $A=\pi R^2$ (trivially; the border plane doesn't matter for the rays that hit the sphere at all)
3) $-R\sin\beta<h<R\sin\beta$. Then there are two parts: the part of the equator and the part of the cross-cut. The equator projects faithfully and the cross cut with the $\cos\beta$ factor. All we need is to find the area of each. The separating line lies at the distance $H=\frac{|h|}{\sin\beta}$ from the center of the equator disk (for $h>0$, we need the small segment of the equator disk and the large segment of the separation disk; for $h<0$, it is the opposite. The length of that line is $2\sqrt{R^2-H^2}$
Now, if we have a disk of radius $r$ and a chord of length $2\ell$, the (small) segment that is cut off has the area $r^2\arcsin\frac\ell r-\ell\sqrt{r^2-\ell^2}$ (sector minus triangle).
Now, let's bring everything together.
If $h\ge 0$, then we get
$$
A=R^2\arcsin\frac {\sqrt{R^2\sin^2\beta-h^2}}{R\sin\beta}-\frac
{h\sqrt{R^2\sin^2\beta-h^2}}{\sin^2\beta}+
\\
\cos\beta\left[(R^2-h^2)\left(\pi-\arcsin\frac{\sqrt{R^2\sin^2\beta-h^2}}{\sin\beta\sqrt{R^2-h^2}}\right) 
+\frac{h\cos\beta\sqrt{R^2\sin^2\beta-h^2}}{\sin^2\beta}\right]\,.
$$
If $h\le 0$, we get
$$
A=R^2\left(\pi-\arcsin\frac {\sqrt{R^2\sin^2\beta-h^2}}{R\sin\beta}\right)
+\frac{|h|\sqrt{R^2\sin^2\beta-h^2}}{\sin^2\beta}+
\\
\cos\beta\left[(R^2-h^2)\arcsin\frac{\sqrt{R^2\sin^2\beta-h^2}}{\sin\beta\sqrt{R^2-h^2}}
-\frac{|h|\cos\beta\sqrt{ R^2\sin^2\beta-h^2}}{\sin^2\beta} \right]\,.
$$
I agree that this expression may make a nervous person pass out and admit that I might make a stupid typo somewhere when $\LaTeX$-ing it, but I believe I wrote enough for you to be able to make an independent derivation. Enjoy! :)
