1. An optimization method of image depth information, adapted for an image processing apparatus, the optimization method comprising: obtaining a to-be-repaired depth map generated based on a left image and a right image, wherein the to-be-repaired depth map records a plurality of first valid depth values and a plurality of holes, wherein the plurality of holes are corresponding to a plurality of invalid depth values and distributed over the to-be-repaired depth map; performing a superpixel segmenting process on one of the left image and the right image to obtain a plurality of superpixels of one of the left image and the right image; aggregating the superpixels according to pixel information in the superpixels to obtain a plurality of image segments; performing a hole filling process on the holes in the to-be-repaired depth map to obtain a hole-filled depth map that comprises a plurality of second valid depth values; using a range divided by the image segments, a range divided by the superpixels, the to-be-repaired depth map, and the hole-filled depth map to perform a statistical analysis on the first valid depth values of the to-be-repaired depth map and the second valid depth values of the hole-filled depth map, so as to obtain a plurality of optimized depth values; and obtaining an optimized depth map according to the optimized depth values.