
For different encoder layers in the baseline model (see “L4 in Base”, “L5 in Base” and “L6 in Base”), lower layers embed more about surface information
while higher layers encode more semantics, which are consistent with previous findings in [1]}.

Integrating the shallow sentence representation (“+ Ssr”) obtains improvement over the baseline on semantic tasks (75.33 vs. 74.61), while fails to improve on the surface (77.32 vs. 77.60) and syntactic tasks (64.88 vs. 65.00). This may indicate that the shallow representations that exploits only the top encoder layer (“L6 in Base”) encodes more semantic information.

Introducing deep sentence representation (“+ Dsr”) brings more improvements. The reason is that our deep sentence representation is induced from the sentence representations of all the encoder layers, and lower layers that contain abound surface and syntactic information are exploited.

