You might not even realize that the X-Men not being with the Avengers is even a whole thing, but it is. Before it was a stand-alone production studio, in the 1990s Marvel was just a bankrupt comic publisher . In order to stay afloat, Marvel started selling off the rights to some of its most famous characters — so the X-Men went to Fox, Spider-Man went to Sony, and for a while Iron Man was actually at New Line Cinemas (before going to Paramount where the first Iron Man movie was made). And thus, the characters in the Marvel universe separated into their own separate universes on film.