In this paper, we address the challenge of limited labeled data and class imbalance problem for machine learning-based rumor detection on social media. We present an offline data augmentation method based on semantic relatedness for rumor detection. To this end, unlabeled social media data is exploited to augment limited labeled data. A context-aware neural language model and a large credibility-focused Twitter corpus are employed to learn effective representations of rumor tweets for semantic relatedness measurement. A language model fine-tuned with the a large domain-specific corpus shows a dramatic improvement on training data augmentation for rumor detection over pretrained language models. We conduct experiments on six different real-world events based on five publicly available data sets and one augmented data set. Our experiments show that the proposed method allows us to generate a larger training data with reasonable quality via weak supervision. We present preliminary results achieved using a state-of-the-art neural network model with augmented data for rumor detection. Research areas that have recently been received much attention in using Machine Learning (ML) and Natural Language Processing for automated rumor and fake news detection BID5 BID11 BID19 BID25 BID22 and fact-checking BID2 BID21 BID10 . One major bottleneck of state-of-the-art (SoA) ML methods is that they require a vast amount of labeled data to be trained and manual labeling of rumors source on social media requires special skills and time-consuming BID26 . Due to limited labeled training data, existing neural networks (NNs) for rumor detection usually have shallow architecture BID3 BID13 . The scarcity of labeled data is a major challenge of studying rumors on social media BID0 . Another problem is that publicly available data sets for rumor-related tasks such as PHEME data BID10 suffer from imbalanced class distributions . Existing methods for handling the class imbalance problem (e.g., oversampling and the use of synthetic data BID24 ) may cause over-fitting and poor generalization performance. A methodology for rumor data augmentation with the minimum of human supervision is necessary. Previous studies presented that rumors can evolve into many variants which share similar propagation patterns in their early stage BID14 BID3 BID1 BID4 . Based on this hypothesis, we argue that enriching existing labeled data with unlabeled source tweets conveying the same or similar meanings is a promising attempt for rumor detection methods that rely on the structure of rumor propagation in social media. In this work, we propose a novel data augmentation method for automatic rumor detection based on semantic relatedness. We exploit a publicly available paraphrase identification corpus as well as context-sensitive embeddings of labeled references and unlabeled candidate source tweets. Pairwise similarity is used to guide the assignment of pseudolabels to unlabeled tweets. ELMo BID18 , a SoA context-sensitive neural language model (NLM), is fine-tuned on a large credibility-focused social media corpus and used to encode tweets. Our results show that data augmentation can contribute to rumor detection with deep learning with increased training data size and a reasonable level of quality. This has potential for further performance improvements using deeper NNs. We present data augmentation results for three events and the performance of a SoA DNN model for rumor detection with augmented data in Section 5. Data Augmentation Before filtering out source tweets without replies, 1,238 rumors and 3,714 non-rumors are collected for "bostonbombings". After filtering, 165 rumors and 228 non-rumors remain. Although the augmented data size is very limited for "bostonbombings", experiments on "sydneysiege" and "ottawashooting" show encouraging results. A total of 25,486 rumors and 76,106 non-rumors are additionally obtained for "sydneysiege", and 21,519 rumors and 62,590 non-rumors are additionally obtained for "ottawashooting". We make our augmented data publicly available 4 . Rumor Detection We conduct rumor detection experiments using two different data sets: (1) PHEME5, (2) PHEME5 with the "bostonbombings" data ("PHEME5+Boston"). We employ BID10 's method as a SoA baseline model for rumor detection with slight modifications. For the sake of simplicity, we modify the implementation of "MTL2 Veracity+Detection" for rumor detection only. We construct input by using a source tweet and the top (i.e., most recent) 24 replies in this task. We perform leave-one-out cross-validation (LOOCV) on the PHEME5 and augmented data sets. The overall experimental results for rumor detection are presented in TAB4 . TAB5 shows LOOCV results. We observe that overall performance decreases with the augmented data (i.e., PHEME5+Boston). The "fergusonunrest" is the most difficult event for a rumor detection model as it has a unique class distribution distinguished from all other events BID10 . It is worth noting that our data augmentation improves the performance of rumor detection on the "fergusonunrest". The completion of data augmentation for events other than "'bostonbombings" has potential to boost overall and per event performance of rumor detection. We present a methodology of data augmentation for rumor detection that exploits semantic relatedness between limited labeled data and unlabeled data. This study is part of further research that aims to use a massive amount of publicly available unlabeled Twitter data and the potential of DNNs in a wide range of tasks related to rumors on social media. Our current research has demonstrated the potential efficiency and effectiveness of semantically augmented data in combating the labeled data scarcity and class imbalance problems of publicly available rumor data sets. In future work, we plan to augment data for more events to build comprehensive data sets for rumor detection, and conduct experiments on rumor detection via deep learning. We will evaluate the effectiveness of augmented data in alleviating over-fitting and its usefulness in facilitating deeper NNs for rumor detection. Further experiments will be conducted to examine the generalization of rumor detection models on unseen rumors.
We propose a methodology of augmenting publicly available data for rumor studies based on samantic relatedness between limited labeled and unlabeled data.
{"src": "scitldr-AIC_train", "paper_id": "SyxCysRNdV"}