1. A system for localizing and controlling a tool tip with a robotic arm, the system comprising: one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of: receiving three-dimensional (3D) visual information of a scene having a tool with a tool tip proximate a target; generating a depth disparity map based on the 3D visual information; filtering the depth disparity map by depth continuity to generate loose boundaries of the tool; shrinking and extracting the boundaries of the tool as an extracted tool boundary; identifying ends of the tool tip based on the extracted tool boundary; and generating a pose of the tool tip.