In this paper, we combine (1) the robot affordance model of <cite>[10]</cite> , which associates verbal descriptions to the physical interactions of an agent with the environment, with (2) the gesture recognition system of [4] , which infers the type of action from human user movements.