Q:

Why “modding out the homeomorphism” in the category Top makes no rigorous sense?

We can rigorously talk about Top, the category of all topological space, and also FTop, the category of all finite topological space. So I thought, we can define a category FTop', where we “mod out by homeomorphism” between objects in FTop. i.e. define the equivalence relation ~ in FTop as X ~ Y if they are homeomorphic and define FTop’ as FTop/~. But recently I am told that this “makes no rigorous sense”. Why is that?

A:

There are a couple of things to watch out for.
First, the collection of all spaces which are homeomorphic to a given space $X$ forms a proper class, and so we cannot just naively use set theory to form it. Nevertheless this is not a true obstacle, since we can employ Scott's trick.
Second, two spaces can be homeomorphic in many ways, which presents a problem when we try to define morphisms between equivalence classes. Writing $[X]$ for the homeomorphism class of $X$, what could the morphisms $[X] \to [Y]$ be? One might naively try to answer with "equivalence classes of continuous maps $f : X \to Y$ of some sort". However, this will not work so easily. Here's an obvious attemp that does not work. Given $X \cong X'$ and $Y \cong Y'$ and $f : X \to Y$ and $f' : X' \to Y'$, say that $f \cong f'$ if there are homeomorphisms $p : X \to X'$ and $q : Y \to Y'$ such that $f' \circ p = q \circ f$ (as suggested in the comments). Then composition of functions is not a congruence and so composition of morphisms is unclear (as noted in the comments).
As far as I can tell, the problem persists with finite spaces as well. If fact, the problem is not specific to topological spaces at all. As soon as there can be multiple isomorphisms between two objects, we'll have a problem.
One possible solution (which I initially called "standard" but people in the comments questioned it) is not to quotient the category but to pass to its skeleton, i.e., we form the full subcategory on chosen representatives of homeomorphism classes. In general this may require some form of the axiom of choice, but let's not worry about that today.
Supplemental: The discussion in the comments veered in the direction of homotopy type theory, so it might be worth looking at what happens there. If we take topological spaces to live at the level of 0-types (also known as h-sets), then by the structure identity principle homeomorphic spaces are going to be equal. This may seem mistifying at first, for does such a statement not suffer from the same defect regarding morphisms, as above? No, because HoTT is proof-relevant. Concretely, suppose $h : X' \to X$ is a homeomorphism and $f : X \to Y$ is a continuous map. The following does not work in HoTT:

Faulty HoTT reasoning: Because $X'$ and $X$ are homeomorphic, by the structure identity principle $X' = X$, and since $f : X \to Y$ then also $f : X' \to Y$.

We must pay attention to why $X' = X$ (that's what proof relevance is about, proofs are mathematical objects, they're not just published stories and logicians' fetishes):

Because $h : X' \to X$ is a homeomorphism, by the structure identity principle $\mathsf{sip}(h)$ proves $X = X'$. We may therefore transport any construction involving $X$ to one on $X'$ along $\mathsf{sip}(h)$. In particular, given $f : X \to Y$, there is $\mathsf{transport}_{\mathsf{sip}(h)}(f) : X' \to Y$. A little calculation reveals that $\mathsf{transport}_{\mathsf{sip}(h)}(f) = f \circ h$.

Note how instead of saying "isomorphic things are equal" we make things proof-relevant by saying "every isomorphism begets a proof of equality".

