What are the Differences Between a Matrix and a Tensor? What is the difference between a matrix and a tensor? Or, what makes a tensor, a tensor? I know that a matrix is a table of values, right? But, a tensor?
Maybe to see the difference between rank 2 tensors and matrices, it is probably best to see a concrete example. Actually this is something which back then confused me very much in the linear algebra course (where we didn't learn about tensors, only about matrices).
As you may know, you can specify a linear transformation $a$ between vectors by a matrix. Let's call that matrix $A$. Now if you do a basis transformation, this can also be written as a linear transformation, so that if the vector in the old basis is $v$, the vector in the new basis is $T^{-1}v$ (where $v$ is a column vector). Now you can ask what matrix describes the transformation $a$ in the new basis. Well, it's the matrix $T^{-1}AT$.
Well, so far, so good. What I memorized back then is that under basis change a matrix transforms as $T^{-1}AT$.
But then, we learned about quadratic forms. Those are calculated using a matrix $A$ as $u^TAv$. Still, no problem, until we learned about how to do basis changes. Now, suddenly the matrix did not transform as $T^{-1}AT$, but rather as $T^TAT$. Which confused me like hell: how could one and the same object transform differently when used in different contexts?
Well, the solution is: because we are actually talking about different objects! In the first case, we are talking about a tensor that takes vectors to vectors. In the second case, we are talking about a tensor that takes two vectors into a scalar, or equivalently, which takes a vector to a covector.
Now both tensors have $n^2$ components, and therefore it is possible to write those components in a $n\times n$ matrix. And since all operations are linear resp. bilinear, the normal matrix-matrix and matrix-vector products together with transposition can be used to write the operations of the tensor. Only when looking at basis transformations, you see that both are, indeed, not the same, and the course did us (well, at least me) a disservice by not telling us that we are really looking at two different objects, and not just at two different uses of the same object, the matrix.
Indeed, speaking of a rank-2 tensor is not really accurate. The rank of a tensor has to be given by two numbers. The vector to vector mapping is given by a rank-(1,1) tensor, while the quadratic form is given by a rank-(0,2) tensor. There's also the type (2,0) which also corresponds to a matrix, but which maps two covectors to a number, and which again transforms differently.
The bottom line of this is:

The components of a rank-2 tensor can be written in a matrix.
The tensor is not that matrix, because different types of tensors can correspond to the same matrix.
The differences between those tensor types are uncovered by the basis transformations (hence the physicist's definition: &quot;A tensor is what transforms like a tensor&quot;).

Of course, another difference between matrices and tensors is that matrices are by definition two-index objects, while tensors can have any rank.
"Tensors are generalizations of scalars (that have no indices), vectors (that have exactly one index), and matrices (that have exactly two indices) to an arbitrary number of indices. " [1]

You can't index into a scalar $s\in \mathbb{R}$. You can index into a vector $v\in \mathbb{R}^d$ with a single index $i$ as $v_i$. You can index into a matrix $M\in \mathbb{R}^{n \times m}$ with two indices $(i, j)$ as $M_{ij}$. 

If I continue adding indices, you might imagine a matrix-like object $T\in\mathbb{R}^{n\times m \times p}$ you can index into with three indices $(i, j, k)$ as $T_{ijk}$. How about $4, 5, 6, ...$ indices? A tensor is such an object that allows any number of indices you want. 

In this way scalars, vectors and matrices are the special case of tensors with 0, 1 and 2 indices. 

[1] http://mathworld.wolfram.com/Tensor.html