Action-conditional video prediction has been used to grasp and sweep with novel tools [1]} in a visual MPC framework [2]}.
TOG-Net [3]} trains a task-oriented grasp policy based on DexNet [4]} by predicting the probability of task success (such as hammering or sweeping) given a grasp pose.
A separate tool use policy is trained with the policy gradient algorithm. KETO [5]} learns to generate keypoints from point clouds and uses quadratic programming for tool control. Grasping is dependent on outputs from a pretrained GraspNet [6]} model.
GIFT [7]} also learns to generate keypoints for tool use but is also dependent on a separate DexNet-like model for grasping. Lastly, other works  [8]} have shown that a combination of differentiable physics and hierarchical mixed-integer planning can optimize complex tool use trajectories without detailed reward shaping, followed up by recent work expanding its applicability [9]}.
<FIGURE>