I feel like a lot of the issue is that there is a huge disconnect in the three "main" ways that trap is used.  It's used in 1) Porn 2) Anime/Manga 3) Towards real people. In the third way it's super offensive, because like you said it's implying trans women aren't real women. In the first two however, it's not always referring to trans women. While I'm sure that there are trans women in "trap" porn, often times they are male identifying who aim for a feminine look. In that case the setting of the porn itself is that they are men trying to "trick" men, so it's an accurate description. Finally, when it comes to anime/manga the characters are also male identifying, outside of the (extremely rare) series where characters are transexual, and in those those issues are often addressed.