1. A hearing assistance system for receiving and processing sounds with control of directionality of sound reception, comprising: a processor configured to process the received sounds and including an adaptive directionality controller configured to control a target direction of the sound reception using a beamformer configured to cancel sound signals that are not coming from the target direction, the sound signals including off-axis speech and noise, the adaptive directionality controller configured to: detect the off-axis speech; and steer the beamformer to reduce the cancellation of the off-axis speech without substantially changing the cancellation of the noise in response to a detection of the off-axis speech.