tokenizer issue while trying the hindi model
#1
by
samruddhakf
- opened
I used the code given in model card tab for an audio. I am getting an output which is basically not using any vowel modification of consonant. Abugida features are not getting used. For example: For an audio, I am getting output like --- अर अब ज न भ द ज उसक नह म लम थ when it should be अरे अब जाने भी दीजिए, उसको नहीं मालूम था
what I am missing, how can I fix this? Is there particular parameter I need to set?
Thanks!