As a passenger, if the pilot doesn't want to fly for whatever reason, I don't want to go.  As you have pointed out, it's always better to be safe on the ground somewhere than to take unnecessary risks in the air.  Thanks to all pilots who exercise good judgement and aren't afraid to say "no".