We found only three lines of work that specifically looked into Bengali readability [6] , <cite>[11]</cite> , [21] . Around the same time, Islam et al. independently reached the same conclusion <cite>[11]</cite> . While all the above studies are very important and insightful, none of them explicitly performed an inter-rater agreement study. Further, none of these studies made available their readability-annotated gold standard datasets, thereby stymieing further research. We attempt to bridge these gaps in our work.