The characteristic that sets them apart from the invertebrates is the presence of a spinal column that is usually surrounded by a backbone or vertebral column.
Backbone is another name for the vertebral column.