How can a hom-set have a group structure? I'm trying to understand the definition of an Ab-enriched category, but I don't get how a hom-set can have a group structure. Doesn't $Hom_C(a, b)$ consist only of morphisms of the form $f: a \rightarrow b$? How does then composition of morphisms in the group work when the domain and codomain don't match? Is the group operation not morphism composition or is there something I am missing here?
Indeed, the group operation is not composition of morphisms.  It is just some group operation, which is specified as part of the $Ab$-enrichment.  In typical examples, it is the operation of "pointwise addition of homomorphisms".  For instance, if $A$ and $B$ are abelian groups (so our category is $Ab$ itself), then the set $\operatorname{Hom}_{Ab}(A,B)$ of homomorphisms $A\to B$ has a natural abelian group structure: if $f,g:A\to B$ are homomorphisms, then the function $f+g$ defined by $(f+g)(x)=f(x)+g(x)$ is also a homomorphism.  This group structure on $\operatorname{Hom}_{Ab}(A,B)$ for each $A,B$ makes $Ab$ into an $Ab$-enriched category.
I believe you are studying the notion of group object in a category $C$; $G$ is a group object if $Hom(X,G)$ has a structure of group and $X\rightarrow Hom(X,G)$ is a contravariant functor to the category of groups.

Suppose for example $C$ is the category of sets, and $G$ is a group, you endow $Hom(X,G)$ with the structure defined by $(f.g)(x)=f(x).g(x)$. The neutral is the constant map $f(x)=1_G$ and the inverse of $f$ is $(f^{-1})(x)=f(x)^{-1}$.