Background: My wife and I have a pre-teen daughter who has made a new friend via a 'social club' for lack of a better term.  Both girls happen to enjoy a particular activity so my wife and I have taken them twice to said activity.  Today the other child's parents decided to take them to an event and we dropped our daughter off at their house this morning.  
Situation: This afternoon they came to our house to drop off our daughter.  They have an autistic son who as soon as the door opened, stepped into our place and the father, despite saying 'So sorry, we don't mean to intrude', stepped into the room right behind his son.  My wife is in the middle of planting seeds (very long growth tropical plants), so the kitchen was a mess, there were dirty dishes on the counter, there are seeding pots and dirt everywhere, and I was in the middle of hanging shelves so there were tools and dust and parts.  I was congenial while they were here, but after they'd gone I muttered half to myself half to my wife 'I can't stand it when people walk in without being expressly invited, it's fucking rude.'  She became defensive and ended up feeling sad because I was offended and irate (not at her, at the parent of the other kid).  I tried to comfort her, explaining my perspective, that unless expressly invited I would not consider stepping in to someone's house, and unless the visit was expected and planned I would assume the invitation was pro forma and still not enter unless I either had a long term relationship with the family, or the invitation was repeated.  This situation was neither; there was no implied invitation until after they were already in and my wife said 'please come in' nor was there a planned visit.  
This particular situation was complicated by his son who is, I'd venture to say, autistic, and the kid led the way in.  I've lived for 10 years next to a severely autistic child who had been taught to not wander into houses uninvited, and whose parents bent over backwards to prevent their child from being an inconvenience to the neighborhood, and in doing so earned the welcome and earnest support of all of us.  
I dunno, I still feel like what the father allowed his son, and then in turn, him, to do was very rude and I don't think I am wrong for being perturbed.  
Am I the asshole?