Unidirectional models are commonly used in the previous approaches [2, 3, 4, 5, 6,<cite> 7]</cite> . However, this constraint on model architectures limits the potential of speech representation learning.