Following<cite> Chiu and Nichols (2016)</cite>, we used different symbols for word-level capitalization feature each assigned a randomly initialized embedding: allCaps, upperInitial, lowercase, mixedCaps and noinfo.