Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Cloud Computing has become another buzzword after Web 2.0. However, there are dozens of different definitions for Cloud Computing and there seems to be no consensus on what a Cloud is. On the other hand, Cloud Computing is not a completely new concept; it has intricate connection to the relatively new but thirteen-year established Grid Computing paradigm, and other relevant technologies such as utility computing, cluster computing, and distributed systems in general. This paper strives to compare and contrast Cloud Computing with Grid Computing from various angles and give insights into the essential characteristics of both. <s> BIB001 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> This work presents a performance comparison of the two latest video coding standards H.264/MPEG-AVC and H.265/MPEG-HEVC (High-Efficiency Video Coding) as well as the recently published proprietary video coding scheme VP9. According to the experimental results, which were obtained for a whole test set of video sequences by using similar encoding configurations for all three examined representative encoders, H.265/MPEG-HEVC provides significant average bit-rate savings of 43.3% and 39.3% relative to VP9 and H.264/MPEG-AVC, respectively. As a particular aspect of the conducted experiments, it turned out that the VP9 encoder produces an average bit-rate overhead of 8.4% at the same objective quality, when compared to an open H.264/MPEG-AVC encoder implementation - the x264 encoder. On the other hand, the typical encoding times of the VP9 encoder are more than 100 times higher than those measured for the x264 encoder. When compared to the full-fledged H.265/MPEG-HEVC reference software encoder implementation, the VP9 encoding times are lower by a factor of 7.35, on average. <s> BIB002 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Nowadays, multimedia is considered to be the biggest big data as it dominates the traffic in the Internet and mobile phones. Currently symmetric encryption algorithms are used in IoT but when considering multimedia big data in IoT, symmetric encryption algorithms incur more computational cost. In this paper, we have designed and developed a resource-efficient encryption system for encrypting multimedia big data in IoT. The proposed system takes the advantages of the Feistel Encryption Scheme, an Advanced Encryption Standard (AES), and genetic algorithms. To satisfy high throughput, the GPU has also been used in the proposed system. This system is evaluated on real IoT medical multimedia data to benchmark the encryption algorithms such as MARS, RC6, 3-DES, DES, and Blowfish in terms of computational running time and throughput for both encryption and decryption processes as well as the avalanche effect. The results show that the proposed system has the lowest running time and highest throughput for both encryption and decryption processes and highest avalanche effect with compared to the existing encryption algorithms. To satisfy the security objective, the developed algorithm has better Avalanche Effect with compared to any of the other existing algorithms and hence can be incorporated in the process of encryption/decryption of any plain multimedia big data. Also, it has shown that the classical and modern ciphers have very less Avalanche Effect and hence cannot be used for encryption of confidential multimedia messages or confidential big data. The developed encryption algorithm has higher Avalanche Effect and for instance, AES in the proposed system has an Avalanche Effect of %52.50. Therefore, such system is able to secure the multimedia big data against real-time attacks. <s> BIB003 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Mobile users typically have high demand on localized and location-based information services. To always retrieve the localized data from the remote cloud, however, tends to be inefficient, which motivates fog computing. The fog computing, also known as edge computing, extends cloud computing by deploying localized computing facilities at the premise of users, which prestores cloud data and distributes to mobile users with fast-rate local connections. As such, fog computing introduces an intermediate fog layer between mobile users and cloud, and complements cloud computing toward low-latency high-rate services to mobile users. In this fundamental framework, it is important to study the interplay and cooperation between the edge (fog) and the core (cloud). In this paper, the tradeoff between power consumption and transmission delay in the fog-cloud computing system is investigated. We formulate a workload allocation problem which suggests the optimal workload allocations between fog and cloud toward the minimal power consumption with the constrained service delay. The problem is then tackled using an approximate approach by decomposing the primal problem into three subproblems of corresponding subsystems, which can be, respectively, solved. Finally, based on simulations and numerical results, we show that by sacrificing modest computation resources to save communication bandwidth and reduce transmission latency, fog computing can significantly improve the performance of cloud computing. <s> BIB004 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Internet of Things (IoT) is now transitioning from theory to practice. This means that a lot of data will be generated and the management of this data is going to be a big challenge. To transform IoT into reality and build upon realistic and more useful services, better resource management is required at the perception layer. In this regard, Fog computing plays a very vital role. With the advent of Vehicular Ad hoc Networks (VANET) and remote healthcare and monitoring, quick response time and latency minimization are required. However, the receiving nodes have a very fluctuating behavior in resource consumption especially if they are mobile. Fog, a localized cloud placed close to the underlying IoTs, provides the means to cater such issues by analyzing the behavior of the nodes and estimating resources accordingly. Similarly, Service Level Agreement (SLA) management and meeting the Quality of Service (QoS) requirements also become issues. In this paper, we devise a methodology, referred to as MEdia FOg Resource Estimation (MeFoRE), to provide resource estimation on the basis of service give-up ratio, also called Relinquish Rate (RR), and enhance QoS on the basis of previous Quality of Experience (QoE) and Net Promoter Score (NPS) records. The algorithms are implemented using CloudSim and applied on real IoT traces on the basis of Amazon EC2 resource pricing. <s> BIB005 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> The Internet of Things (IoT) opens a new horizon for network applications by virtue of wireless sensor network technologies and advancement of data analytics. Since various applications are possible in the IoT compared to conventional multimedia applications, quality of experience (QoE), which is a user's subjective satisfaction with the application, becomes more complicated for network operators to perceive. In this paper, we propose a framework of scalable QoE modeling for explosively increasing applications. In the framework, the massive amount of quality metrics in the IoT architecture are defined as physical metrics and organized into four layers; device, network, computing, and user interface. We also introduce metaphysical metrics, which are the summarized quality metrics that users of IoT applications demand. By aggregating the massive amount of physical metrics into certain metaphysical metrics and qualitatively modeling the QoE by the metaphysical metrics according to the characteristics of the applications, scalable modeling of the QoE for continuously emerging applications in the IoT becomes possible. <s> BIB006 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> With the popularity of multimedia sensing at cloud edges and the reducing cost of the Internet of Things (IoT) fog devices and systems, new challenges have been posed to efficiently deal with the big data multimedia traffic generated from IoT sensing units. Specifically, in this paper we introduce the concept of multimedia sensing as a service (MSaaS), and propose a generalized premium prioritization-based quality of experience paradigm for wireless big-volumes-of-data (BVDs) multimedia communications, with significant energy saving potentials for future multimedia IoT devices and systems. The key contribution of this new framework is its data diversity flexibility at the application layer, which could be flexibly adopted by future multimedia communication systems. Data dependencies in spatial, frequency and temporal domains are analyzed, and interaction with uplink resource allocation optimization are investigated with regards to wireless communication energy cost estimation. Extensive simulation results demonstrate that the proposed prioritization-based communication paradigm has significant energy saving potentials for BVD MSaaS wireless multimedia communications at cloud edges and fogs. <s> BIB007 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> We investigate the design and implementation of Where's The Bear (WTB), an end-to-end, distributed, IoT system for wildlife monitoring.WTB implements a multi-tier (cloud, edge, sensing) system that integrates recent advances in machine learning based image processing to automatically classify animals in images from remote, motion-triggered camera traps.We use non-local, resource-rich, public/private cloud systems to train the machine learning models, and ``in-the-field,'' resource-constrained edge systems to perform classification near the IoT sensing devices (cameras).We deploy WTB at the UCSB Sedgwick Reserve, a 6000 acre site for environmental research and use it to aggregate, manage, and analyze over 1.12M images.WTB integrates Google TensorFlow and OpenCV applications to perform automatic classification and tagging for a subset of these images.To avoid transferring large numbers of training images for TensorFlow over a low-bandwidth network linking Sedgwick to the public/private clouds, we devise a technique that uses stock Google Images to construct a synthetic training set using only a small number of empty, background images from Sedgwick.Our system is able to accurately identify bears, deer, coyotes, and empty images and significantly reduces the time and bandwidth requirements for image transfer, as well as end-user analysis time, since WTB automatically filters the images on-site. <s> BIB008 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Voluminous amounts of data have been produced, since the past decade as the miniaturization of Internet of things (IoT) devices increases. However, such data are not useful without analytic power. Numerous big data, IoT, and analytics solutions have enabled people to obtain valuable insight into large data generated by IoT devices. However, these solutions are still in their infancy, and the domain lacks a comprehensive survey. This paper investigates the state-of-the-art research efforts directed toward big IoT data analytics. The relationship between big data analytics and IoT is explained. Moreover, this paper adds value by proposing a new architecture for big IoT data analytics. Furthermore, big IoT data analytic types, methods, and technologies for big data mining are discussed. Numerous notable use cases are also presented. Several opportunities brought by data analytics in IoT paradigm are then discussed. Finally, open research challenges, such as privacy, big data mining, visualization, and integration, are presented as future research directions. <s> BIB009 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Event processing systems serve as a middleware between the Internet of Things (IoT) and the application layer by allowing users to subscribe to events of interest. Due to the increase of multimedia IoT devices (i.e. traffic camera), the types of events created are shifting more toward unstructured (multimedia) data. Therefore, there is a growing demand for efficient utilization of effective processing of streams of both structured events (i.e. sensors) and unstructured multimedia events (i.e. images, video, and audio). However, current event processing engines have limited or no support for unstructured event types. In this paper, we described a generalized approach that can handle Internet of Multimedia Things (IoMT) events as a native event type in event processing engines with high efficiency. The proposed system extends event processing languages with the introduction of operators for multimedia analysis of unstructured events and leverages a deep convolutional neural network based event matcher for processing image events to extract features. Furthermore, we show that neural network based object detection models can be further optimized by leveraging subscription constraints to reduce time complexity while maintaining competitive accuracy. Our initial results demonstrate the feasibility of a generalized approach toward IoMT-based event processing. Application areas for generalized event processing include traffic management, security, parking, and supervision activities to enhance the quality of life within smart cities. <s> BIB010 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Abstract With rapid evolution of the Internet of Things (IoT) applications on multimedia, there is an urgent need to enhance the satisfaction level of Multimedia IoT (MIoT) network users. An important and unsolved problem is automatic optimization of Quality of Experience (QoE) through collecting/managing/processing various data from MIoT network. In this paper, we propose an MIoT QoE optimization mechanism leveraging data fusion technology, called QoE optimization via Data Fusion (QoEDF). QoEDF consists of two steps. Firstly, a multimodal data fusion approach is proposed to build a QoE mapping between the uncontrollable user data with the controllable network-related system data. Secondly, an automatic QoE optimization model is built taking fused results, which is different from the traditional way. QoEDF is able to adjust network-related system data automatically so as to achieve optimized user satisfaction. Simulation results show that QoEDF will lead to significant improvements in QoE level as well as be adaptable to dynamic network changes. <s> BIB011 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> As multimedia services are gaining considerable popularity in the Internet of Things, providing QoE in smart mmIoT becomes a challenging issue. This article conceptualizes a new framework named SMP, to price the QoE rather than the binary data traffic in IoT multimedia services. By introducing PaaR in QoE-driven network scheduling in mmIoTs, a new dimension of price distortion is available in multimedia resource allocation, complementing traditional rate distortion and power distortion resource allocation approaches. This article also discusses open research challenges in SMP resource allocation, including the difficulty of numerical presentation of QoE, the unavailability of packet-level distortion information, and the non-concavity of the utility functions. A case study is presented to illustrate the PaaR potentials of improving QoE through the SMP framework in mmIoTs. <s> BIB012 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> Abstract Recently, there has been an increasing interest in the Internet of Things (IoT). While some analysts disvalue the IoT hype, several technology leaders, governments, and researchers are putting serious efforts to develop solutions enabling wide IoT deployment. Thus, the huge amount of generated data, the high network scale, the security and privacy concerns, the new requirements in terms of QoS, and the heterogeneity in this ubiquitous network of networks make its implementation a very challenging task. SDN, a new networking paradigm, has revealed its usefulness in reducing the management complexities in today's networks. Additionally, SDN, having a global view of the network, has presented effective security solutions. On the other hand, fog computing, a new data service platform, consists of pushing the data to the network edge reducing the cost (in terms of bandwidth consumption and high latency) of “big data” transportation through the core network. In this paper, we critically review the SDN and fog computing-based solutions to overcome the IoT main challenges, highlighting their advantages, and exposing their weaknesses. Thus, we make recommendations at the end of this paper for the upcoming research work. <s> BIB013 </s> Multimedia Internet of Things: A Comprehensive Survey <s> F. SUMMARY AND INSIGHTS <s> High Efficiency Video Coding (HEVC) encryption, which has been proposed to encrypt intra prediction modes (structural information), transform coefficients (texture information), and motion related codewords (motion information), has received considerable attention recently. However, there is still the issue of efficiency when HEVC encryption is applied in the Internet of Multimedia Things (IoMT). Aiming at this challenge, in this paper, we propose a new low-overhead HEVC encryption scheme for energy-constrained IoMT. Concretely, the proposed scheme adjusts the selection of the aforementioned syntax elements to be encrypted according to the structure, texture, and motion energy present in each frame. It works as follows. The energy levels of quantized coefficients and motion vectors are calculated and compared with adaptive threshold values to classify the energy level in each video frame. When there is a high energy frame in the video, all the syntax elements are encrypted. When there is a low energy frame, alternate syntax elements are encrypted for achieving low encryption overhead. Moreover, in the case of transform coefficients, to withstand the interpolation attack, alternate coefficients are encrypted after correlating the frame with its neighboring coefficients. Extensive experiments were conducted, and the results demonstrate that the proposed scheme efficiently reduces the encryption overhead with low impact on the security level, making it suitable for IoMT. <s> BIB014
In this section, the limitations of underlying M-IoT architectures for multimedia communication has been discussed. Various multimedia encoding and compression techniques to reduce the complexity of multimedia data transmission in IoT has been detailed in BIB002 ,  - BIB003 . Detection, extraction, and event processing of meaningful information from multimedia data has been studied in BIB014 , BIB010 -  for efficient network resource utilization. The vision of cloud computing and various studies to analysis and process multimedia data over the Internet through remote access is present in BIB001 - BIB007 . Increasing network overheads and the latency due to cloud computing has been discussed, the concept of fog\edge computing to minimize network latency for real-time applications is discussed in BIB004 - BIB008 . The notion of SDNs to enhance the scalability and interoperability to support heterogeneous M-IoT devices has been explained. Various studies related to SDN based fog\edge-cloud computing to reduce computing latency and overheads have been presented in BIB011 , BIB012 , BIB005 , BIB006 , BIB013 ,  ,  - BIB009 . SDNs, not only provides reconfigurability but also offers a more scalable and interoperable architecture. A tabular summarization of this section can be seen in Table 7 .