Previous research used multi-modal information independently using neural network model by concatenating features from each modality<cite> [7,</cite> 21] . As opposed to this approach, we propose a neural network architecture that exploits information in each modality by extracting relevant segments of the speech data using information from the lexical content (and vice-versa).