1. A method of selecting a candidate vector in a multiple input multiple output (MIMO) system, the method comprising: selecting a plurality of candidate vectors corresponding to a first layer among a plurality of layers; sequentially ranking a plurality of constellation dots corresponding to a second layer that is different from the first layer among the plurality of layers for each of the candidate vectors corresponding to the first layer; selecting a plurality of arbitrary constellation dots from the constellation dots ranked for each of the candidate vectors; and selecting candidate vectors corresponding to the second layer on the basis of a first arbitrary constellation dot having a minimal accumulated cost among the plurality of arbitrary constellation dots.