In (relatively) simple words: What is an inverse limit? I am a set theorist in my orientation, and while I did take a few courses that brushed upon categorical and algebraic constructions, one has always eluded me.

The inverse limit. I tried to ask one of the guys in my office, and despite a very shady explanation he ended up muttering that "you usually take an already known construction."

The Wikipedia article presents two approaches, the algebraic and the categorical. While the categorical is extremely vague for me, the algebraic one is too general and the intuition remains hidden underneath the text in a place I cannot find it.

Since I am not too familiar with categories, the explanation most people would try to give me which is categorical in nature seems to confuse me - as I keep asking this question over and over every now and then.

Could anyone explain to me in non-categorical terms what is the idea behind an inverse limit? (I am roughly familiar with its friend "direct limit", if that helps)

(While editing, I can say that the answers given so far are very interesting, and I have read them thoroughly, although I need to give it quite some thinking before I can comment on all of them right now.)
I like George Bergman's explanation (beginning in section 7.4 of his Invitation to General Algebra and Universal Constructions).

We start with a motivating example. 

Suppose you are interested in solving $x^2=-1$ in $\mathbb{Z}$. Of course, there are no solutions, but let's ignore that annoying reality for a moment.

We use the notation $\mathbb{Z}_n$ for $\mathbb Z / n \mathbb Z$.
The equation has a solution in the ring $\mathbb{Z}_5$ (in fact, two: both $2$ and $3$, which are the same up to sign). So we want to find a solution to $x^2=-1$ in $\mathbb{Z}$ which satisfies $x\equiv 2 \pmod{5}$. 

An integer that is congruent to $2$ modulo $5$ is of the form $5y+2$, so we can rewrite our original equation as $(5y+2)^2 = -1$, and expand to get
$25y^2 + 20y = -5$.

That means $20y\equiv -5\pmod{25}$, or $4y\equiv -1\pmod{5}$, which has the unique solution $y\equiv 1\pmod{5}$. Substituting back we determine $x$ modulo $25$:
$$x = 5y+2 \equiv 5\cdot 1 + 2  = 7 \pmod{25}.$$

Continue this way: putting $x=25z+7$ into $x^2=-1$ we conclude $z\equiv 2 \pmod{5}$, so $x\equiv 57\pmod{125}$. 

Using Hensel's Lemma, we can continue this indefinitely. What we deduce is that there is a sequence of residues, 
$$x_1\in\mathbb{Z}_5,\quad x_2\in\mathbb{Z}_{25},\quad \ldots, x_{i}\in\mathbb{Z}_{5^i},\ldots$$
each of which satisfies $x^2=-1$ in the appropriate ring, and which are "consistent", in the sense that each $x_{i+1}$ is a lifting of $x_i$ under the natural homomorphisms
$$\cdots \stackrel{f_{i+1}}{\longrightarrow} \mathbb{Z}_{5^{i+1}} \stackrel{f_i}{\longrightarrow} \mathbb{Z}_{5^i} \stackrel{f_{i-1}}{\longrightarrow}\cdots\stackrel{f_2}{\longrightarrow} \mathbb{Z}_{5^2}\stackrel{f_1}{\longrightarrow} \mathbb{Z}_5.$$

Take the set of all strings $(\ldots,x_i,\ldots,x_2,x_1)$ such that $x_i\in\mathbb{Z}_{5^i}$ and $f_i(x_{i+1}) = x_i$, $i=1,2,\ldots$. This is a ring under componentwise operations. What we did above shows that in this ring, you do have a square root of $-1$. 


Added. Bergman here inserts the quote, "If the fool will persist in his folly, he will become wise." We obtained the sequence by stubbornly looking for a solution to an equation that has no solution, by looking at putative approximations, first modulo 5, then modulo 25, then modulo 125, etc. We foolishly kept going even though there was no solution to be found. In the end, we get a "full description" of what that object must look like; since we don't have a ready-made object that satisfies this condition, then we simply take this "full description" and use that description as if it were an object itself. By insisting in our folly of looking for a solution, we have become wise by introducing an entirely new object that is a solution.

This is much along the lines of taking a Cauchy sequence of rationals, which "describes" a limit point, and using the entire Cauchy sequence to represent this limit point, even if that limit point does not exist in our original set. 



This ring is the $5$-adic integers; since an integer is completely determined by its remainders modulo the powers of $5$, this ring contains an isomorphic copy of $\mathbb{Z}$.

Essentially, we are taking successive approximations to a putative answer to the original equation, by first solving it modulo $5$, then solving it modulo $25$ in a way that is consistent with our solution modulo $5$; then solving it modulo $125$ in a way that is consistent with out solution modulo $25$, etc.

The ring of $5$-adic integers projects onto each $\mathbb{Z}_{5^i}$ via the projections; because the elements of the $5$-adic integers are consistent sequences, these projections commute with our original maps $f_i$. So the projections are compatible with the $f_i$ in the sense that for all $i$, $f_i\circ\pi_{i+1} = \pi_{i}$, where $\pi_k$ is the projection onto the $k$th coordinate from the $5$-adics.

Moreover, the ring of $5$-adic integers is universal for this property: given any ring $R$ with homomorphisms $r_i\colon R\to\mathbb{Z}_{5^i}$ such that $f_i\circ r_{i+1} = r_i$, for any $a\in R$ the tuple of images $(\ldots, r_i(a),\ldots, r_2(a),r_1(a))$ defines an element in the $5$-adics. The $5$-adics are the inverse limit of the system of maps
$$\cdots\stackrel{f_{i+1}}{\longrightarrow}\mathbb{Z}_{5^{i+1}}\stackrel{f_i}{\longrightarrow}\mathbb{Z}_{5^i}\stackrel{f_{i-1}}{\longrightarrow}\cdots\stackrel{f_2}{\longrightarrow}\mathbb{Z}_{5^2}\stackrel{f_1}{\longrightarrow}\mathbb{Z}_5.$$

So the elements of the inverse limit are "consistent sequences" of partial approximations, and the inverse limit is a way of taking all these "partial approximations" and combine them into a "target object."

More generally, assume that you have a system of, say, rings, $\{R_i\}$, indexed by an directed set $(I,\leq)$ (so that for all $i,j\in I$ there exists $k\in I$ such that $i,j\leq k$), and a system of maps $f_{rs}\colon R_s\to R_r$ whenever $r\leq s$ which are "consistent" (if $r\leq s\leq t$, then $f_{rs}\circ f_{st} = f_{rt}$), and let's assume that the $f_{rs}$ are surjective, as they were in the example of the $5$-adics. Then you can think of the $R_i$ as being "successive approximations" (with a higher indexed $R_i$ as being a "finer" or "better" approximation than the lower indexed one). The directedness of the index set guarantees that given any two approximations, even if they are not directly comparable to one another, you can combine them into an approximation which is finer (better) than each of them (if $i,j$ are incomparable, then find a $k$ with $i,j\leq k$). The inverse limit is a way to combine all of these approximations into an object in a consistent manner.

If you imagine your maps as going right to left, you have a branching tree that is getting "thinner" as you move left, and the inverse limit is the combination of all branches occurring "at infinity". 



Added. The example of the $p$-adic integers may be a bit misleading because our directed set is totally ordered and all maps are surjective. In the more general case, you can think of every chain in the directed set as a "line of approximation"; the directed property ensures that any finite number of "lines of approximation" will meet in "finite time", but you may need to go all the way to "infinity" to really put all the lines of approximation together. The inverse limit takes care of this. 

If the directed set has no maximal elements, but the structure maps are not surjective, it turns out that no element that is not in the image will matter; essentially, that element never shows up in a net of "successive approximations", so it never forms part of a "consistent system of approximations" (which is what the elements of the inverse limit are).
I'm currently dabbling around in a category where morphisms are refinements of computer programs, in the sense that they are functions $f : X \rightarrow Y$ from a program refinement (an implementation) $X$ to a program specification $Y$. 

In this setting, a sequence (or net) of morphisms can be either describing an ever more elaborate specification $X_0 \rightarrow X_1 \rightarrow X_2 \rightarrow ...$ or it can be an ever more precise refinement $... \rightarrow X_2 \rightarrow X_1 \rightarrow X_0$. The first sequence has as a direct limit the 'ultimate' specification, while the second sequence has as an inverse (also called projective) limit the 'ultimate' refinement.

As an extra: maybe my own question about Directed and projective limit in Rel helps in looking at the notion in a different way. In Rel, where the two types of limit coincide, my intuition is that a net of relations simply results in 'lines' connecting points in a disjoint union, and the 'infinite' lines form a new object, the limit.