We pretrain CLSRIL-23 model [1]} on 23 Indic languages using wav2vec 2.0 architecture[2]} on 10000 hours of audio data. It learns the cross lingual speech representations from raw waveforms. For ASR pretraining, audio chunks with a duration of 25s-30s are used. Pretraining is started with a learning rate of 0.0005 and dropout of 0.1 using Adam optimizer for the first 32,000 steps. After this, the learning rate decreases linearly and the model is trained for almost 300,000 steps.
