1. A method for hierarchical motion vector processing, comprising: receiving a motion vector field for a block in an image and residual energy concerning the image block; developing a motion vector reliability classification using the residual energy; developing a merging map based upon the residual energy distribution and vector reliability classification and merging blocks based upon the merging map; selecting motion vectors for each merged group of blocks from its own and neighboring block reliable motion vectors; determining the bi-direction difference between the reference frame and the decoded frame for the selected motion vectors; re-classifying motion vector reliability using the bi-directional difference; refining motion vectors between the reference frame and current decoded frame using the re-classified reliability and the bi-directional difference; and smoothing motion vectors.