I think the main problems are that Dean frequently gets storylines where he is portrayed as special...and then these are dropped.
Sam gets storylines where we are Told he is special, to the point of it being shoved down our throats, but we are never Shown it.