In this section, we discuss our implementation of the neural model AutoCoder to address the automatic code generation problem. Recently, Transformers [1]} have shown state-of-the-art performance for several Natural Language Processing tasks [2]}, [3]}, [4]}, including machine translation, classification, etc. Inspired by its success, we propose a transformer-based code-generation model to generate code based on natural language descriptions automatically. Specifically, the model encodes the textual description using multiple layers of encoders and then decodes a program token-by-token while attending to the description. The basic pipeline of our proposed model is analogous to the simpler sequence-to-sequence models that are employed for similar generation tasks. These models usually have an encoder-decoder architecture [5]}, [6]}. The encoder maps an input sequence of symbol representations consisting of tokens \(x_1,x_2,\cdots ,x_n\)  to intermediate representation which the decoder then generates an output sequence \(y_1,y_2,\cdots ,y_m\)  of symbols one element at a time. At each step, the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next output symbol. As depicted in Figure REF , we utilize the core Transformer [1]} model implementation and propose significant structural alterations in the attention module to develop AutoCoder.
<FIGURE>