Ratio of balls in a box A box contains some identical tennis balls. The ratio of the total volume of the tennis
balls to the volume of empty space surrounding them in the box is $1:k$, where $k$ is an
integer greater than one.
A prime number of balls is removed from the box. The ratio of the total volume of the
remaining tennis balls to the volume of empty space surrounding them in the box is 1:$k^2$.
Find the number of tennis balls that were originally in the box.
A few questions regarding this problem:
Does the shape of the box matter? I just let the volume of the box be a constant $V$. Also I noted that the ratio $\frac{1}{k^2} = \left( \frac{1}{k} \right)^2$. ie. New ratio is the old ratio squared. I also let the amount of balls in the box be $n$ and the amount of balls taken out be $p$ where $p$ is a prime, so the new amount of balls in the box is $n-p$.
This is about all I could do in this problem but I would like to be guided towards solving the problem (and I'm also interested in your thought processes and what ideas you initially think of so I can get a better idea of what to think of when doing problem solving) than just being given the solution.
Your help would be much appreciated.

The ratio $1:k$ means in particular that if $\alpha$ is the total volume of the balls originally in the box, then $k\alpha$ is the total volume of the empty space surrounding them in the box. In general, if $V_t$ is the total volume of the tennis balls and $V_b$ is the total volume of the box, then the total volume of empty space surrounding the tennis balls in the box is $V_b-V_t.$ (This has nothing to do with the shape of the box. We simply need the box to be large enough that it can hold all the tennis balls originally in it, and still be closed.) What this means is that the total volume of the box (which will not be changing) is $\alpha+k\alpha=(1+k)\alpha.$
Now, let's suppose that there are $n$ balls in the box originally, and since they are all tennis balls, then we can assume that they all have the same volume. (In fact, if they had different volumes, there would be no way to do the problem, so we need to assume it.) Say that the volume of a single tennis ball is $\beta.$ Then $\alpha=n\beta,$ so the volume of the box is $(1+k)n\beta.$
Next, we remove a prime number of balls from the box, say $p,$ so that there are now $n-p$ balls in the box, so that the total volume of the tennis balls remaining is $(n-p)\beta.$ The total volume of the box is $(1+k)n\beta,$ so the total volume of the empty space around the remaining tennis balls in the box is $$(1+k)n\beta-(n-p)\beta=n\beta+kn\beta-n\beta+p\beta=(kn+p)\beta.$$ By assumption, then, the ratio $1:k^2$ is the same as the ratio $(n-p)\beta:(kn+p)\beta,$ which is clearly the same as $n-p:kn+p.$ This means that $$kn+p=k^2(n-p)\\kn+p=k^2n-k^2p\\k^2p+p=k^2n-kn\\(k^2+1)p=(k^2-k)n.$$ Now, note that since we removed a prime (so non-zero) number of balls from the box, then our new ratio cannot be the same as our old one--that is, $k^2\ne k$--so we may divide by $k^2-k$ to get $$n=\frac{k^2+1}{k^2-k}p.$$ At this point, I don't think there's much more we can say, unless there's some additional information about $k,p,$ or $n$ that you haven't shared. We can conclude readily that $\frac{k^2+1}{k^2-k}$ is a rational number greater than $1,$ but that doesn't really help to determine what $n$ has to be (at least, not as far as I can see).

Oops! I missed the condition that $k$ was an integer! Ignore "Now, note that since we removed...as far as I can see)."
Now, since $k$ is an integer, then $k^2+1$ and $k^2-k$ are integers. In particular, since $$(k^2+1)p=k(k-1)n,$$ then $k\mid(k^2+1)p,$ so since $k$ and $k^2+1$ are relatively prime, then we must have that $k\mid p.$ Since $p$ is prime and $k$ is an integer greater than $1,$ it then follows that $k=p,$ so we have $$(k^2+1)p=(k-1)np\\k^2+1=(k-1)n\\k^2+1=kn-n\\k^2-kn+n+1=0.$$ This gives us a quadratic in $k,$ whose solutions are $$\begin{align}k &= \frac{n\pm\sqrt{n^2-4(n+1)}}2\\ &= \frac{n\pm\sqrt{n^2-4n-4}}2.\end{align}$$ Since $k$ is an integer, then we require $\sqrt{n^2-4n-4}$ to be rational, which means it must be a positive integer, say $m.$ Hence, $$n^2-4n-4=m^2\\n^2-4n+4-8=m^2\\(n-2)^2-8=m^2\\(n-2)^2-m^2=8\\\bigl((n-2)+m\bigr)\bigl((n-2)-m\bigr)=8\\(n-2+m)(n-2-m)=8.$$ Since $m$ is a positive integer, then we can conclude that $m=1,$ whence $n=5.$ (I leave it to you to show this.)
