Recent progress on deep learning has made it possible to automatically transform the screenshot of Graphic User Interface (GUI) into code by using the encoder-decoder framework. While the commonly adopted image encoder (e.g., CNN network), might be capable of extracting image features to the desired level, interpreting these abstract image features into hundreds of tokens of code puts a particular challenge on the decoding power of the RNN-based code generator. Considering the code used for describing GUI is usually hierarchically structured, we propose a new attention-based hierarchical code generation model, which can describe GUI images in a finer level of details, while also being able to generate hierarchically structured code in consistency with the hierarchical layout of the graphic elements in the GUI. Our model follows the encoder-decoder framework, all the components of which can be trained jointly in an end-to-end manner. The experimental results show that our method outperforms other current state-of-the-art methods on both a publicly available GUI-code dataset as well as a dataset established by our own.
To tackle the above issue, @cite_7 employ the hierarchically structured LSTMs, and their model is able to generate paragraph-level image descriptions. Basically, in their work, two levels of LSTM-based language decoders are used: a first-stage LSTM captures the general information of the image and stores context information of each sentence in its hidden states. Then, a second-stage LSTM is used to decode the first-stage LSTM's hidden states into different sentence in a paragraph. Our method for automatic GUI code generation also employs a similar hierarchical approach. However, in our method, first-stage LSTM's hidden states are only used in the attention network to help select the CNN features, which are then fed into the second-stage LSTM for code generation. Our experimental results show that the first-stage LSTM's hidden states, which only contain a coarse level of visual information, are insufficient for guiding the second-stage LSTM to generate accurate token sequences. We do obtain better results by feeding the more information-carrying raw CNN features into the second-stage LSTM.
@cite_7
2549599535
Recent progress on image captioning has made it possible to generate novel sentences describing images in natural language, but compressing an image into a single sentence can describe visual content in only coarse detail. While one new captioning approach, dense captioning, can potentially describe images in finer levels of detail by captioning many regions within an image, it in turn is unable to produce a coherent story for an image. In this paper we overcome these limitations by generating entire paragraphs for describing images, which can tell detailed, unified stories. We develop a model that decomposes both images and paragraphs into their constituent parts, detecting semantic regions in images and using a hierarchical recurrent neural network to reason about language. Linguistic analysis confirms the complexity of the paragraph generation task, and thorough experiments on a new dataset of image and paragraph pairs demonstrate the effectiveness of our approach.