For the input descriptions, we replace all punctuations with a space; all letters are lower cased. For the neural network, we set the number of CNN layers \(L\)  to 21, where the bottom layer does not have skipping connections. We also find it helpful to build a separate network (i.e., the same architecture, but with different weight parameters) for different AST node types (namely, nonterminal nodes, variable nodes, and function name nodes). This enables us to be better aware of node types during generation. When predicting variable nodes, we introduce a new softmax target, for each slot, that could copy the slot value. The layers of difference CNN modules are set to the same dimension, chosen by validation from {128, 192, 256} for each predictor network. We applied dropout (drop rate\(=0.5\) ) and \(\ell _2\)  penalty to regularize the fully connected layers.
The network is trained by the Adam optimizer [1]} with default hyperparameters.
<TABLE>