1. A method of decoding video data, the method comprising: searching a set of neighbor blocks of the video data to generate a plurality of disparity vector candidates, wherein each of the neighbor blocks is a neighbor of a current block of the video data, wherein generating the plurality of disparity vector candidates comprises storing disparity motion vectors or implicit disparity vectors (IDVs) of the neighbor blocks as the plurality of disparity vector candidates; in response to determining that all of the disparity vector candidates have the same reference view index, applying a mathematical function to the plurality of disparity vector candidates to determine a final disparity vector for the current block, wherein the mathematical function is one selected from a group consisting of a median operation, a minimum function, and a maximum function; in response to determining that not all of the disparity vector candidates have the same reference view index, determining that the final disparity vector for the current block is one of the disparity vector candidates; and reconstructing a sample block for the current block in part by performing, based on the final disparity vector for the current block, inter-view motion prediction or inter-view residual prediction for the current block.