What is the link between homomorphisms and mutual information? Intuitively, there seems to be a link between the (kind of) homomorphism between two algebraic structures and the mutual information between two variables. However, since I'm not a mathematician, it's hard to see or formalize the exact connection. Anyone who can help me with this?

For example, if we'd take each variable to represent a set and its order relation, could we say that the very existence of a homomorphic map between the two would make their $ MI \gt 0 $ ?

And the other way around: would an $ MI = 0 $ between those variables make the map between them a random one?
Let's look at two examples.

In the case of vector spaces we have linear maps, or vector space homomorphisms. These preserve scalar products (1) and are linear(2).

Let $f: V \to W$ be a linear map.

$$f\big(\lambda x \in V \big) = \lambda f(x) \in W\quad (1)$$
$$f\big(x+y \in V\big) = f(x) + f(y) \in W \quad (2)$$

So we see that the operation of addition and scalar multiplication is preserved between $V$ and $W$.

In the case of groups, we have group homomorphisms. Similarly these preserve group operations.

Let $g: A \to B$ be a group homomorphism and $*$ the group operation.

$$g\big(a_1 * a_2 \in A\big) = g(a_1) * g(a_2) \in B.$$

So, in a sense, we can say the "mutual information" of the operations is preserved when a homomorphism maps an element from one object to another.
Let's look at two examples.

In the case of vector spaces we have linear maps, or vector space homomorphisms. These preserve scalar products (1) and are linear(2).

Let $f: V \to W$ be a linear map.

$$f\big(\lambda x \in V \big) = \lambda f(x) \in W\quad (1)$$
$$f\big(x+y \in V\big) = f(x) + f(y) \in W \quad (2)$$

So we see that the operation of addition and scalar multiplication is preserved between $V$ and $W$.

In the case of groups, we have group homomorphisms. Similarly these preserve group operations.

Let $g: A \to B$ be a group homomorphism and $*$ the group operation.

$$g\big(a_1 * a_2 \in A\big) = g(a_1) * g(a_2) \in B.$$

So, in a sense, we can say the "mutual information" of the operations is preserved when a homomorphism maps an element from one object to another.