Pullback of a set is its preimage (posting this as a question-answer combo, as figuring out this relation helped me understand the terms, and I hope it might for others)
I have heard that the preimage of a function is sometimes referred to the pullback.
Specifically, given a map $f:A\to B$ and a subset $E\subseteq B$, that the preimage of $E$ under $f$, $f^{-1}(E)$, may also be called the pullback of $E$ under the map $f$, $f^*E$.
I have encountered the notion of a pullback several times in geometry where, for example, the pullback of a map $g:B\to C$ under a map $f:A\to B$ may be given as $f^*(g) = g\circ f$. Is there a relationship between these two concepts?

Pullbacks are a much more general idea: one of the most general framings is in category theory. The category theory diagram (see nLab) for a pullback shows why the name is apt (it is like you are "pulling back" two arrows with the same endpoint, to a common starting point).
A pullback in the sense of a set preimage $f^{-1}(B)$ is, in categorical terms, a pullback of a function $f:A\rightarrow B$ and the identity $\text{id}_B:B\rightarrow B$ in the category $\mathbf{Set}$. (See Wikipedia.)
The pullbacks you have encountered in geometry are likely all categorical pullbacks. However, some controversy is had over whether viewing everything in category theory terms is useful.
