What does this physics paper mean by having a matrix in a denominator? I am not familiar with any notation used in physics. The paper "Non-hermitian random matrix theory: Method of hermitian reduction" by 
Joshua Feinberg and A. Zee (Nuclear Physics B, 1997) states:

A basic tool in studying hermitian random matrices $\phi$ (henceforth all matrices will
  be taken to be $N\times N$ with $N$ tending to infinity unless otherwise specified) is the Green
  function defined by
  $$ G(z) = \left< \frac{1}{N} \text{tr} \frac{1}{z - \phi}\right>,$$
  where $\langle \dots \rangle$ denotes averaging over the random distribution from which the matrices $\phi$ are drawn. Diagonalizing $\phi$ by a unitary transformation we have
  $$G(z) = \left< \frac{1}{N} \sum_{k=1}^N\frac{1}{z - \lambda_k}\right>,$$
  where the $N$ real numbers $\{ \lambda_k \}$ are the eigenvalues of $\phi$.

In the second equation, everything makes sense. But in the first equation, there is the strange notation of a matrix in the denominator of a fraction, and subtracting a matrix from a scalar. Working backwards, it seems like the first equation is supposed to mean $(zI - \phi)^{-1}$. But why is it not written that way?

Your $(zI-\phi)^{-1}$ interpretation is correct. The motivation for avoiding it is for analogy with the scalar/eigenvalue case, where we can freely use $\frac{N}{D}$ notation. This is especially important when we think about inverse operators, e.g. a Green's function (aka propagator) such as $\int\frac{d^4k}{(2\pi)^4}\frac{\exp ik\cdot x}{k^2-m^2+i\epsilon}$.
