The cricket club is concerned as the building is intended to go on land it uses next to its ground, which it has rented from the NHS for around 50 years.
Plans had been proposed to build a primary care centre on the land, known as the hospital field, with the potential closure of Trowbridge Hospital considered.