This work contributes a) an approach to build robust visual classifiers which distinguish verbs, objects, and places extracted from weak sentence annotations; b) based on the visual classifiers we evaluate different design choices to train an LSTM for generating descriptions. This outperforms <cite>related work</cite> on the <cite>MPII-MD dataset</cite>, both using automatic and human evaluation; c) we perform a detailed analysis of prior work and our approach to understand the challenges of the movie description task.