1. A method for decoding video data, the method comprising: performing a disparity vector derivation process to derive a disparity vector for a coding unit (CU) of the video data, wherein the CU is partitioned into a plurality of prediction units (PUs) and each respective PU of the plurality of PUs is associated with a respective motion vector, wherein each of the PUs of the CU shares the derived disparity vector for the CU and the disparity vector derivation process is only invoked once for the CU, wherein performing the disparity vector derivation process comprises: checking one or more neighboring blocks of the CU for a disparity motion vector; and converting the disparity motion vector to the derived disparity vector; and as part of a process to reconstruct the video data, performing, based on the derived disparity vector, inter-view prediction for two or more PUs of the plurality of PUs.