The lower voltage means higher current for a given amount of electrical power delivered to the starter.
As chemistry teaches us, higher current will wear out the solenoid contacts quicker due to metal transfer from the higher current.