Multimedia Internet of Things: A Comprehensive Survey <s> B. EVENT PROCESSING <s> Event processing systems serve as a middleware between the Internet of Things (IoT) and the application layer by allowing users to subscribe to events of interest. Due to the increase of multimedia IoT devices (i.e. traffic camera), the types of events created are shifting more toward unstructured (multimedia) data. Therefore, there is a growing demand for efficient utilization of effective processing of streams of both structured events (i.e. sensors) and unstructured multimedia events (i.e. images, video, and audio). However, current event processing engines have limited or no support for unstructured event types. In this paper, we described a generalized approach that can handle Internet of Multimedia Things (IoMT) events as a native event type in event processing engines with high efficiency. The proposed system extends event processing languages with the introduction of operators for multimedia analysis of unstructured events and leverages a deep convolutional neural network based event matcher for processing image events to extract features. Furthermore, we show that neural network based object detection models can be further optimized by leveraging subscription constraints to reduce time complexity while maintaining competitive accuracy. Our initial results demonstrate the feasibility of a generalized approach toward IoMT-based event processing. Application areas for generalized event processing include traffic management, security, parking, and supervision activities to enhance the quality of life within smart cities. <s> BIB001 </s> Multimedia Internet of Things: A Comprehensive Survey <s> B. EVENT PROCESSING <s> Multimedia Sensor Networks (MSNs) have gained much attention in recent years from the emerging trends of Internet of Things (IoT). They can be found in different scenarios in our everyday life (e.g., smart homes, smart buildings). Sensors in MSNs can have different capacities, produce multiple kinds of outputs, and have different output encoding formats. Thus, detecting complex events, which requires the aggregation of several sensor readings, can be difficult due to the lack of a generic model that can describe: (i) sensor networks infrastructure, (ii) individual sensor specificities, as well as (iii) multimedia data, while allowing the alignment with the application domain knowledge. In this study, we propose Multimedia Semantic Sensor Network Ontology (MSSN-Onto) to ensure MSNs modeling and provide both syntactic and semantic data interoperability for defining and detecting events in various domains. To show the readiness of MSSN-Onto, we used it as the core ontology of a dedicated framework (briefly defined here). We also adopted MSSN-Onto in HIT2GAP European Project. A prototype has been implemented to conduct a set of tests. Experimental results show that MSSN-Onto can be used to: (i) effectively model MSNs and multimedia data; (ii) define complex events; and (iii) allow to build an efficient event querying engine for MSNs. \textcopyright 2018 Elsevier B.V. <s> BIB002 </s> Multimedia Internet of Things: A Comprehensive Survey <s> B. EVENT PROCESSING <s> In this paper, we propose an approach to generate the comprehensive video interpretation for the surveillance video understanding in Internet of Things. The key problem of many visual learning tasks is to adaptively select and fuse diverse and complimentary features for video representation. We design the attention-in-attention (AIA) network to hierarchically explore the attention fusion in an end-to-end manner, and demonstrate the value of this model on the multievent recognition and video captioning challenges. Particularly, it consists of multiple encoder attention modules (EAMs) and a fusion attention module (FAM). Each EAM aims to highlight the space-specific features by selecting the most salient visual features or semantic attributes and averages them into one attentive feature. The FAM can suppress or enhance the activation of multispace attentive features and adaptively co-embed them for comprehensive video representation. Then, one long short-term memory unit decodes the video representations to generate multiple event labels or video captions. This architecture is capable of: 1) adaptively learning the salient space-specific feature representation and 2) co-embedding multispace attentive features into one space for feature fusion. Experiments conducted on the surveillance video dataset (concurrent event dataset) and the popular video captioning datasets (Microsoft Research Video Description Corpus and MSR-Video to Text). It shows that the proposed AIA can achieve competitive performances against the state of the arts. <s> BIB003 </s> Multimedia Internet of Things: A Comprehensive Survey <s> B. EVENT PROCESSING <s> High Efficiency Video Coding (HEVC) encryption, which has been proposed to encrypt intra prediction modes (structural information), transform coefficients (texture information), and motion related codewords (motion information), has received considerable attention recently. However, there is still the issue of efficiency when HEVC encryption is applied in the Internet of Multimedia Things (IoMT). Aiming at this challenge, in this paper, we propose a new low-overhead HEVC encryption scheme for energy-constrained IoMT. Concretely, the proposed scheme adjusts the selection of the aforementioned syntax elements to be encrypted according to the structure, texture, and motion energy present in each frame. It works as follows. The energy levels of quantized coefficients and motion vectors are calculated and compared with adaptive threshold values to classify the energy level in each video frame. When there is a high energy frame in the video, all the syntax elements are encrypted. When there is a low energy frame, alternate syntax elements are encrypted for achieving low encryption overhead. Moreover, in the case of transform coefficients, to withstand the interpolation attack, alternate coefficients are encrypted after correlating the frame with its neighboring coefficients. Extensive experiments were conducted, and the results demonstrate that the proposed scheme efficiently reduces the encryption overhead with low impact on the security level, making it suitable for IoMT. <s> BIB004
The M-IoT encompasses myriad applications in every field which varies in data capacities, features, nature of outputs, VOLUME 8, 2020 and encoding formats. An increasing number of devices and applications transits the nature of multimedia traffic more towards unstructured events. Generic event detection and query processing framework is in need to process structured and unstructured multimedia events. Aslam and Curry BIB001 proposed a Multimedia Stream Processing Engine (MSPE) as a middleware with an operator to analysis unstructured events in multimedia data and DNN processor to extract and match the features. Publish/subscribe mechanism is adapted in which multimedia devices send captured events to the middleware and subscribers can access the desired events by relevant classifiers from middleware by DNN based matchers. Optimization models are adapted to analyze and process a user's query to achieve optimized throughput and accuracy to serve the query. The integration of diverse image analysis techniques could enhance the genericity of the framework. Angsuchotmetee et al. in BIB002 offered Multimedia Semantic Sensor Network Ontology (MSSN-Onto) to detect various events in a HetNet to provide syntactic and semantic interoperability. A multimedia application that involves more than one user requires syntactic and semantic interoperability to understand different data encoding techniques and contentaware modeling. The proposed framework constitutes MSSN indexer, application manager, and event processing engine to process and index the data stream concerning their feature levels, and serving the queries of the users from the stored, indexed data. Xu et al. proposed an attention-in-attention (AIA) network in BIB003 for multievent recognition and representing the salient features of visual modality and semantic modality exploiting Convolutional Neural Network (CNN) for visual features extraction. AIA comprises Encoder Attention Modules (EAM) and Fusion Attention Module (FAM) for desired features matching and fuses multi-dimensional features into a single-dimensional feature. The authors employed a Recurrent Neural Network (RNN) based Long Short-Term Memory (LSTM) unit on decoding fused data into multi-events labels. Multiple AIA modules can enhance event recognition and representations of multimedia video streaming. Saha et al.  presented a context-aware Block-based Motion Estimation (BME) approach to enhance multimedia compression. The author uses the pixel distortion ratio to characterize the motion into a Large Diamond Search Pattern (LDSP) or Small Diamond Search Pattern (SDSP). The proposed scheme intends to reduce the computation time significantly. Parallel processing incorporating GPU and directional motion estimation can further augment the performance. Thiyagarajan et al. BIB004 devised energy-aware selective encryption for the high motion video frame. The detection of high motion is performed by the texture energy level of the frame which is classified by the DC coefficient in Discrete Cosine Transform (DCT) and the motion vector which expresses descriptive information of video magnitude and phase angle. The high value of the motion vector indicates high motion activity and vice versa. For securing multimedia data and conserving energy in M-IoT, all syntax elements of high motion frames are encrypted. However, in low energy frames, alternate syntax elements are encrypted.