Are "$n$ by $n$ matrices with rank $k$" an affine algebraic variety? Identify the set of all complex $n$ by $n$ matrices with $\mathbb{C}^{n^2}$.  We say a subset $S \subset \mathbb{C}^{n^2}$ is an affine algebraic variety if $S$ is the common zero set of a collection (possibly infinite or uncountable) of polynomials in $n^2$ complex variables.  Some examples that satisfy this definition are "matrices with determinant 1" and "matrices with rank at most $k$".  
A non-example is "matrices with rank $n$", because it is the complement of the affine algebraic variety "matrices with rank at most $n - 1$"; any affine algebraic variety is necessarily closed in the Euclidean topology, so its complement (which is open in the Euclidean topology) cannot be an affine algebraic variety.
However, I'm having trouble formulating a statement about "matrices with rank $k$" for $1 < k < n$.

Is the set of complex $n$ by $n$ matrices of rank $k$, where $1 < k < n$, an affine algebraic variety?


At the end of his answer, Pete Clark asks whether the quasi-affine variety of $n \times n$ matrices of rank $k$ might be affine. The answer is no, except when $k=n$.
This is a special case of a much more general fact. Let $X$ be an irreducible affine variety, and let $Y$ be a closed subvariety. If $X \setminus Y$ is affine, then $Y$ is  codimension $1$ in $X$.
Proof Sketch: (This argument is edited in response to points raised to me by Steven Sam.)
Let $r$ be the codimension of some component of $Y$. 
Then the local cohomology group $H^r_Y(X, \mathcal{O})$ is nonzero. 
I earlier said that this was by the inequality between depth and codimension, but Steven points out to me that I am misusing this; that inequality implies that $H^s_Y(X, \mathcal{O})$ is nonzero for some $s \leq r$. I think the inequality I meant to cite is Grothendieck's nonvanishing theorem.
Look at Hartshorne Exercise III.2.2.3.e . There is an exact sequence which, in part, is
$$H^r(X \setminus Y, \mathcal{O}) \to H^{r+1}_Y(X, \mathcal{O}) \to H^{r+1}(X, \mathcal{O}).$$
Since $X$ is affine, the right hand side is zero, and we just claimed that the middle term is nonzero. So $H^r(X \setminus Y, \mathcal{O})$ is nonzero. If $r>1$, this implies that $X \setminus Y$ is non-affine. QED
The rank $k$ matrices have dimension $n^2 - (n-k)^2$. One can check that this is only codimension $1$ for $(k, k-1) = (n,n-1)$.
By the way, it is NOT true that $X$ affine, $Y$ codimension $1$ implies $X \setminus Y$ affine. The simplest counter-example I know is $X = \mathrm{Spec}\ k[w,x,y,z]/(wz-xy)$ and $Y = \{ w=x=0 \}$. 
