Is there a relation that is irreflexive, anti-symmetric and not transitive? from the set $\{a, b, c, d\}$?
Of the one's I have tried, it at best is two of the three, but never all.

If non-transitive means "never transitive for any triple", that is impossible for a complete tournament (irreflexive, never-symmetric relation) with $4$ or more players.  The question sounds like an exercise of rediscovering that fact.  
If  partial tournaments are allowed then it can be done with any number of players, just partition them into 3 categories A,B,C, and have every A player always beat the B players who always beat C who always beat A, and no matches within any category.
