I think in really bad cases, as petty as it sounds, it comes down to an attitude of "my house, my rules."  Believe me, you do not want to get members of the cabin crew ticked off at you.  

Minor disagreements can quickly spiral out of control especially after takeoff with the constant engine noise which makes what could have been a quiet civil conversation impossible.  Throw in the tightly packed passenger situation in the economy sections then you have an environment that can try the patience of the best of us.  (And yes, I've been in seats where there was a fully reclined passenger directly in front of me and a shrieking, constantly kicking the seat back child directly behind - all through the five-hour flight.)  If I had my way, if airlines want to pack economy passengers ever tighter by progressively reducing the seat pitch, then at some point they need to ban reclining seats altogether.  I'm sure that if sardines could talk, they'd complain too.