This paper puts forward a new text to tensor representation that relies on information compression techniques to assign shorter codes to the most frequently used characters. This representation is language-independent with no need of pretraining and produces an encoding with no information loss. It provides an adequate description of the morphology of text, as it is able to represent prefixes, declensions, and inflections with similar vectors and are able to represent even unseen words on the training dataset. Similarly, as it is compact yet sparse, is ideal for speed up training times using tensor processing libraries. As part of this paper, we show that this technique is especially effective when coupled with convolutional neural networks (CNNs) for text classification at character-level. We apply two variants of CNN coupled with it. Experimental results show that it drastically reduces the number of parameters to be optimized, resulting in competitive classification accuracy values in only a fraction of the time spent by one-hot encoding representations, thus enabling training in commodity hardware. Document classification is one of the principal tasks addressed in the context of natural language processing BID22 . It implies associating a document -or any text fragment, for that matter-with a category or label relying on their content. The increasing availability of texts in digital form, especially through the Internet, has called for the development of statistical and artificial intelligence tools for automating this process. Spam detectors, sentiment analysis, news archiving, among many others, demand high-quality text classifiers.There is a broad range of approaches to document classification (see BID22 BID1 BID11 BID15 ). An important portion of them relies on a representation that handles words as the atomic element of text. Consequently, those methods carry out their analysis through statistics of words occurrence . However, the variability of words and structures belonging to a language hinders the viability of this method. That is why, these models have a superior performance in specific domains and applications, where the vocabulary is or can be restricted to a relatively small number of words, possibly chosen by a specialist. Furthermore, such modeling becomes specific to a language, causing the replication process in another language to be carried out from scratch .In recent years, we have experienced a revolution in the machine learning with the advent of deep learning methods BID8 . The development of convolutional neural networks (CNNs) BID17 coupled with the popularization of parallel computing libraries (e. g. Theano BID2 , Tensorflow BID0 , Keras BID7 , etc.) that simplify general-purpose computing on graphics processing units (GPGPU) BID21 has been successful in tackling image classification problem BID16 quickly becoming the state of the art of the field.As it could be expected, the success of deep learning and CNNs in the image classification domain has prompted the interest to extend the deep learning principles to the document classification domain. Some existing methods have been updated but the clear majority are still based on the to-kenization of words and the inference of their statistics. Bag of Words (BoW) BID13 and Word2vec BID20 are some of the most popular strategies.It can be argued that the replication of image classification success in the documents domain faces as main challenge the difficulty of representing text as numerical tensors.To address this issue, suggested a groundbreaking approach that considers the characters as the atomic elements of a text. In particular, they represented the text as a sequence of one-hot encoded characters. This encoding provides a robust, language-independent representation of texts as matrices, that are then used as inputs of different CNNs. Their experimental results showed that this approach was able to attain and, in some cases, improve the state of the art results in complex text classification problems. More recently, BID25 improved those results by combining CNNs with Long Short-Term Memories (LSTMs) BID10 . In spite of that, the impact of this idea is hampered by the large computational demands of the approach, since its training can take days per epoch in relatively complex problems.Character-level representations have the potential of being more robust than word-level ones. On the other hand, they are computationally more expensive because detecting syntactic and semantic relationships at the character-level is more expensive (Blunsom et al., 2017) . One possible solution could be a word representation that incorporates the character-level information.In this paper, we propose an efficient character-level encoding of word to represent texts derived from the Tagged Huffman BID23 information compression technique. This encoding takes into account the character appearance frequency in the texts in order to assign shorter codes to the most frequently used ones. This novel text encoding makes the idea put forward by more computationally accessible by reducing its training requirements in terms of time and memory.The proposed encoding makes possible to represent larger portions of texts in a less sparse form, without any loss of information, while preserving the ability to encode any word, even those not present in the training dataset ones. In order to study the impact of this encoding, we coupled it with two CNN architectures. The experimental studies performed showed that we managed to achieve a performance similar or in some cases better than the state of the art at a fraction of the training time even if we employed a simpler hardware setup.Our main contribution is to show that this novel character-level text encoding produces a reduced input matrix, leading to a substantial reduction in training times while producing comparable or better results in terms of accuracy than the original approach by . This opens the door to more complex applications, the use of devices with lower computational power and the exploration of other approaches that can be coupled with input representation.The rest of the paper is structured as follows. In the next section , we deal with the theoretical foundations and motivation that are required for the ensuing discussions. There we also analyze the alternatives to character-level text compression that were taken into account for producing our proposal. After that, in Section 3, we describe the encoding procedure and the neural network architectures that will take part of the experiments. Subsequently, in Section 4, we replicate the experiments of in order to contrast our proposal with theirs under comparable conditions. Finally, in Section 5, we provide some final remarks, conclusive comments and outline our future work directions.
Using Compressing tecniques to Encoding of Words is a possibility for faster training of CNN and dimensionality reduction of representation
{"src": "scitldr-AIC_train", "paper_id": "SkYXvCR6W"}