Big Data Quality: A systematic literature review and future research directions <s> Context-aware techniques <s> The ability to detect and process anomalies for Big Data in real-time is a difficult task. The volume and velocity of the data within many systems makes it difficult for typical algorithms to scale and retain their real-time characteristics. The pervasiveness of data combined with the problem that many existing algorithms only consider the content of the data source; e.g. a sensor reading itself without concern for its context, leaves room for potential improvement. The proposed work defines a contextual anomaly detection framework. It is composed of two distinct steps: content detection and context detection. The content detector is used to determine anomalies in real-time, while possibly, and likely, identifying false positives. The context detector is used to prune the output of the content detector, identifying those anomalies which are considered both content and contextually anomalous. The context detector utilizes the concept of profiles, which are groups of similarly grouped data points generated by a multivariate clustering algorithm. The research has been evaluated against two real-world sensor datasets provided by a local company in Brampton, Canada. Additionally, the framework has been evaluated against the open-source Dodgers dataset, available at the UCI machine learning repository, and against the R statistical toolbox. <s> BIB001 </s> Big Data Quality: A systematic literature review and future research directions <s> Context-aware techniques <s> Performing predictive modelling, such as anomaly detection, in Big Data is a difficult task. This problem is compounded as more and more sources of Big Data are generated from environmental sensors, logging applications, and the Internet of Things. Further, most current techniques for anomaly detection only consider the content of the data source, i.e. the data itself, without concern for the context of the data. As data becomes more complex it is increasingly important to bias anomaly detection techniques for the context, whether it is spatial, temporal, or semantic. The work proposed in this paper outlines a contextual anomaly detection technique for use in streaming sensor networks. The technique uses a well-defined content anomaly detection algorithm for real-time point anomaly detection. Additionally, we present a post-processing context-aware anomaly detection algorithm based on sensor profiles, which are groups of contextually similar sensors generated by a multivariate clustering algorithm. Our proposed research has been implemented and evaluated with real-world data provided by Powersmiths, located in Brampton, Ontario, Canada. <s> BIB002 </s> Big Data Quality: A systematic literature review and future research directions <s> Context-aware techniques <s> Environmental sensing is becoming a significant way for understanding and transforming the environment, given recent technology advances in the Internet of Things (IoT). Current environmental sensing projects typically deploy commodity sensors, which are known to be unreliable and prone to produce noisy and erroneous data. Unfortunately, the accuracy of current cleaning techniques based on mean or median prediction is unsatisfactory. In this paper, we propose a cleaning method based on incrementally adjusted individual sensor reliabilities, called influence mean cleaning (IMC). By incrementally adjusting sensor reliabilities, our approach can properly discover latent sensor reliability values in a data stream, and improve reliability-weighted prediction even in a sensor network with changing conditions. The experimental results based on both synthetic and real datasets show that our approach achieves higher accuracy than the mean and median-based approaches after some initial adjustment iterations. <s> BIB003 </s> Big Data Quality: A systematic literature review and future research directions <s> Context-aware techniques <s> Ensemble Stream Modeling and Data-cleaning are sensor information processing systems have different training and testing methods by which their goals are cross-validated. This research examines a mechanism, which seeks to extract novel patterns by generating ensembles from data. The main goal of label-less stream processing is to process the sensed events to eliminate the noises that are uncorrelated, and choose the most likely model without over fitting thus obtaining higher model confidence. Higher quality streams can be realized by combining many short streams into an ensemble which has the desired quality. The framework for the investigation is an existing data mining tool. ::: First, to accommodate feature extraction such as a bush or natural forest-fire event we make an assumption of the burnt area (BA*), sensed ground truth as our target variable obtained from logs. Even though this is an obvious model choice the results are disappointing. The reasons for this are two: One, the histogram of fire activity is highly skewed. Two, the measured sensor parameters are highly correlated. Since using non descriptive features does not yield good results, we resort to temporal features. By doing so we carefully eliminate the averaging effects; the resulting histogram is more satisfactory and conceptual knowledge is learned from sensor streams. ::: Second is the process of feature induction by cross-validating attributes with single or multi-target variables to minimize training error. We use F-measure score, which combines precision and accuracy to determine the false alarm rate of fire events. The multi-target data-cleaning trees use information purity of the target leaf-nodes to learn higher order features. A sensitive variance measure such as ƒ-test is performed during each node's split to select the best attribute. Ensemble stream model approach proved to improve when using complicated features with a simpler tree classifier. ::: The ensemble framework for data-cleaning and the enhancements to quantify quality of fitness (30% spatial, 10% temporal, and 90% mobility reduction) of sensor led to the formation of streams for sensor-enabled applications. Which further motivates the novelty of stream quality labeling and its importance in solving vast amounts of real-time mobile streams generated today. <s> BIB004
The last sub-category of Outlier Detection is Context-aware. There are only two papers, which have the same authors, in this sub-category. In BIB001 and BIB002 , there are two main components: first Content Detection and second Context Detection. On Content Detection, each sensor uses its historical data to generate the corresponding regression model, then the data that is far from the model is considered as a content anomaly. Context Detection has two tasks: first, clustering and profiling the sensors, and the second, comparing the content anomaly with the average value of the sensor group. In order to cluster the sensors, the k-means algorithm is used, and the clustering parameters are location, building, ownership, time of year, time of day, and weather phenomena. If the content anomaly is far from the average of each cluster, it is detected as a context anomaly. The difference between the two papers is that in the second paper map-reduce was used to allocate clusters. Evaluation is done on three datasets, and the results show that the proposed method has been able to detect both content and context anomalies in real time. The point is that any content anomaly cannot be a context anomaly. There are three papers in Context-aware, which the first one is a manuscript and the last two are theses. In BIB003 , a value is considered as a predicted value before the sensor generates the data. This value is derived from the mean of the data produced by the sensors correlated with the corresponding sensor. Then the newly generated data is compared to the predicted value, if the difference is less than a threshold, the data is correct and, if more, the predicted value is considered to be the final value. Finally, the sensor's reliability level is also updated. Experiments have been conducted with artificial and real datasets, and also the precision of the method has been evaluated. In this thesis BIB004 , an introduction to data cleaning is first introduced. Then the data stream processing has been investigated. Also, stream processing problems are discussed. Then a cleaning algorithm is presented for the wireless sensor network. For this purpose, when viewing the corrupted and lost data, some k sensors that are associated with the corresponding sensor are chosen, and the final value is taken from the selected values from those sensors. Also, managing mobile trajectories and the effects of noise due to user mobility is considered. In this dissertation  , there is a method for cleaning the data stream of mobility sensors. In this method, a concept called the virtual sensor is used. Since sensors are moving and the location cannot be accurately predicted, virtual static sensors are used to clean the missing values generated by real sensors. To do this, the weighted average of actual sensor values is calculated, and the data value of each virtual sensor is updated so that the weight of the data produced is higher by the sensors near the virtual sensors. Then, with a predictive model, the next virtual sensor data is predicted. If the base station determines that the data produced by the real sensor has an error, refer to the timestamp and spatial range of that sensor, the predicted value of the corresponding virtual sensor is considered as the cleared data. The correct prediction rate and cleaning time are evaluated, and results demonstrated that, when the location information of a sensor is not missing or fully retrieved by a location prediction, this method cleans more than 80% of missing data.