1. A target object holding apparatus comprising: an image obtaining unit configured to obtain an image including a target object; a storing unit configured to store a dictionary for estimating an orientation of the target object; an estimation unit configured to estimate an orientation of the target object included in the image by referring to the dictionary; a controlling unit configured to cause a holding unit to hold the target object based on the orientation of the target object estimated by said estimation unit; a detection unit configured to detect a failure of holding by said holding unit; and a modifying unit configured to modify the dictionary based on the orientation of the target object when said detection unit detects a holding failure.