'Linux' math program with interactive terminal? Are there any open source math programs out there that have an interactive terminal and that work on linux?
So for example you could enter two matrices and specify an operation such as multiply and it would then return the answer or a error message specifying why an answer can't be computed? I am just looking for something that can perform basic matrix operations and modular arithmetic.

In addition to the various heavy-weight tools already named, you should also take a look at Euler Math Toolbox. It is a free, portable, and open source math environment that supports both matrix operations and symbolic manipulation.
It has a straightforward notebook page usage model, with results, plots, and graphics interspersed with the commands and equations that produce them. It is also extensible via Python for those so inclined.
An interesting alternative for some applications is Geogebra. It is an interactive geometry, algebra, and calculus application, intended for teachers and students. 
Another interesting alternative to the well known tools is GSL Shell. This is a tool that wraps the Lua language around the Gnu Scientific Library. It supports a wide array of numeric methods on vectors and scalars. Being based on Lua, the syntax is distinctly different in flavor from the MATLAB inspired Octave or any of the tools based on Python. But that flavor might be to your liking.
