Sorry for the wall of text.

My husband and I have been married about a year and a half and together for 4. He is currently 29 and I am 26. My husband has quite a large friend group that I have met over the years. We have pretty close relationships with several of his friends and others that we have maintained a more of an aquantaince relationship with. Due to us living in the same area he grew up in, most of these relationships were from his younger/college years.

About a month ago we were at a party for one of his friends that he went to high school/college with. The party itself was fine and everyone was very nice. Most of the people there I had not met yet so he went and introduced me to a bunch of his olds friends. We ended up talking with one couple in particular for a majority of the time and they seemed really nice and nothing felt out of place or weird. My husband and the other husband said we should all get together sometime to hang out.

This weekend we finally had dinner with them this past weekend. Nothing overly weird or awful happened but I kept noticing the wife giving my husband odd looks and looking vaguely uncomfortable when we were talking one on one and making offhand comments that alluded to her knowing my husband better than me which I didn't pick up at the time but when I thought about it later on that night it seemed like weird things to say. 

So, Sunday night I come out and ask my husband if they maybe had dated during high school/college. He said no, you know I didn't really date much at this time (he has told me previously that he slept around with pretty much anyone and didn't seriously date until he was 23). Naturally I ask if they have had a relationship outside of dating and he explains that he had a pretty steady physical relationship that continued for about 4 years when she moved for a new job.

At that time I no longer felt the need/want to continue hanging out with this couple. My husband got confused and wanted to know what was wrong. I explained that I didn't want us to maintain a relationship with someone he had been intimate for so long with. He thought that was dumb and made a comment to the fact that she didn't seem to have an issue with it. I argued that based off of her reaction, she possibly did have an issue with it. I also said that I was surprised her husband even wanted us to hang out, and that is when my husband said that he doesn't think her husband actually knows about their relationship. 

This snowballed into a huge fight and I said that it wasn't up for discussion and he called me crazy and insecure for assuming anything would happen between the 2 of them now. AITA?

Tldr: hang out with a couple my husband went to school with, found out later that he slept with the wife and I no longer want us to maintain a relationship with them. 