1. A method of detecting a transmission signal from a received signal in a multiple input multiple output (MIMO) system, the method comprising: obtaining a unitary matrix Q and an upper triangular matrix R by performing a sorted QR-decomposition (SQRD) algorithm with respect to a matrix B indicating a channel state; calculating a vector y by multiplying a transpose matrix Q t of the unitary matrix Q by the received signal Y; dividing the upper triangular matrix R into a plurality of sub-upper triangular matrices and dividing the calculated vector y into a plurality of sub-vectors so as to correspond to the divided plurality of sub-upper triangular matrices; and detecting a lattice point corresponding to each of the divided sub-vectors using the divided plurality of sub-upper triangular matrices.