He probably considers them to be deplorables:  they don't have enough money to shower on him, nor enough clout to organize fancy lunches.  The high life is all Gardner is interested in.  Being a "public servant" is just his front.