In this paper, we propose to scale up discriminative training of<cite> (He and Deng, 2012)</cite> to train features with 150 million parameters, which is one order of magnitude higher than previously published effort, and to apply discriminative training to redistribute probability mass that is lost due to model pruning.