Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> Continuous monitoring activities in surveillance system generate massive amount of data to be transferred from Internet-of-Things (IoT) devices (i.e., cameras and sensors) to a centralized processing unit for analysis. However, several issues need to be addressed, including data migration over bandwidth limited and high latency communication networks, and the heterogeneous nature of data obtained from the surveillance system. The aim of this paper is to develop a scalable and lightweight intelligent distributed surveillance system that make use of an integrated framework of Internet-of-Things (IoT) and cloud computing. It leverages on the pervasiveness in IoT and ubiquitous property of cloud computing technology. This paper introduces the concept of bringing cloud closer to the IoT in order to address the resource poverty of IoT and the dependency for massive data transmission to distant cloud. For providing real-time on-site object detection, this paper instantiates a cloudlet on resource-rich nearby infrastructure which is connected to the IoT devices for distributed retrieval and processing of critical and sensitive data. <s> BIB001 </s> Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> Audio/visual recognition and retrieval applications have recently garnered significant attention within Internet-of-Things-oriented services, given that video cameras and audio processing chipsets are now ubiquitous even in low-end embedded systems. In the most typical scenario for such services, each device extracts audio/visual features and compacts them into feature descriptors, which comprise media queries. These queries are uploaded to a remote cloud computing service that performs content matching for classification or retrieval applications. Two of the most crucial aspects for such services are: 1) controlling the device energy consumption when using the service, and 2) reducing the billing cost incurred from the cloud infrastructure provider. In this paper, we derive analytic conditions for the optimal coupling between the device energy consumption and the incurred cloud infrastructure billing. Our framework encapsulates: the energy consumption to produce and transmit audio/visual queries, the billing rates of the cloud infrastructure, the number of devices concurrently connected to the same cloud server, the query volume constraint of each cluster of devices, and the statistics of the query data production volume per device. Our analytic results are validated via a deployment with: 1) the device side comprising compact image descriptors (queries) computed on Beaglebone Linux embedded platforms and transmitted to Amazon Web Services (AWS) Simple Storage Service, and 2) the cloud side carrying out image similarity detection via AWS Elastic Compute Cloud (EC2) instances, with the AWS Auto Scaling being used to control the number of instances according to the demand. <s> BIB002 </s> Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> People monitoring and tracking activities in surveillance system usually generate massive amount of data from Internet-of-Things (IoT) devices such as cameras. Several issues need to be addressed, including data migration over limited bandwidth and high latency in communication networks. This paper presents an initiative to develop a decentralized face recognition scheme for distributed surveillance system that make use of an integrated framework of Internet-of-Things (IoT) and cloud computing. The decentralized face recognition approach implements a two-stage procedure, including face detection and extraction and face matching. Face detection and extraction are performed on a cloudlet that is located close to the surveillance cameras, hence minimizing the need for massive data transfer to the remote processing center. On the other hand, face matching process is carried out on the face feature vector within a private cloud environment. A case study conducted on the effectiveness of the proposed scheme in detecting “missing” person indicates that the procedure works effectively in the IoT-Cloud infrastructure. <s> BIB003 </s> Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> Wearable and mobile devices are widely used for crowdsensing, as they come with many sensors and are carried everywhere. Among the sensing data, videos annotated with temporal–spatial metadata contain huge amount of information, but consume too much precious storage space. In this paper, we solve the problem of optimizing cloud-based video crowdsensing in three steps. First, we study the optimal transcoding problem on wearable and mobile cameras. We propose an algorithm to optimally select the coding parameters to fit more videos at higher quality on wearable and mobile cameras. Second, we empirically investigate the throughput of different file transfer protocols from wearable and mobile devices to cloud servers. We propose a real-time algorithm to select the best protocol under diverse network conditions, so as to leverage the intermittent WiFi access. Last, we look into the performance of cloud databases for sensor-annotated videos, and implement a practical algorithm to search videos overlapping with a target geographical region. Our measurement study on three popular opensource cloud databases reveals their pros and cons. The three proposed algorithms are evaluated via extensive simulations and experiments. The evaluation results show the practicality and efficiency of our algorithms and system. For example, our proposed transcoding algorithm outperforms existing approaches by 12 dB in video quality, 87% in energy saving, and one-quarter in delivery delay. Another example is, by intelligently choosing a proper cloud database, our system may reduce the insertion time by up to one-third, or the lookup time by up to one-fourth. <s> BIB004 </s> Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> We present OpenFace, our new open-source face recognition system that approaches state-of-the-art accuracy. Integrating OpenFace with inter-frame tracking, we build RTFace, a mechanism for denaturing video streams that selectively blurs faces according to specified policies at full frame rates. This enables privacy management for live video analytics while providing a secure approach for handling retrospective policy exceptions. Finally, we present a scalable, privacy-aware architecture for large camera networks using RTFace. <s> BIB005 </s> Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> We propose an analytical framework for service oriented computing in Internet of Thing scenarios to derive an optimal performance from an elastic extensible cloud that can serve a request from access, edge and core network. If the number of replicas at different levels (access networks, edge networks and core networks) are maintained according to the values derived from our analysis, then the drop rate and resource utilisation improves significantly without compromising the QoS constraints of the service requests. Under this design, the service consumers devices receive the services from the nearest level of replica(if any) at minimal level of cost which is the most rational form of expectation. <s> BIB006 </s> Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> Internet of Things (IoT) systems produce great amount of data, but usually have insufficient resources to process them in the edge. Several time-critical IoT scenarios have emerged and created a challenge of supporting low latency applications. At the same time cloud computing became a success in delivering computing as a service at affordable price with great scalability and high reliability. We propose an intelligent resource allocation system that optimally selects the important IoT data streams to transfer to the cloud for processing. The optimization runs on utility functions computed by predictor algorithms that forecast future events with some probabilistic confidence based on a dynamically recalculated data model. We investigate ways of reducing specifically the upload bandwidth of IoT video streams and propose techniques to compute the corresponding utility functions. We built a prototype for a smart squash court and simulated multiple courts to measure the efficiency of dynamic allocation of network and cloud resources for event detection during squash games. By continuously adapting to the observed system state and maximizing the expected quality of detection within the resource constraints our system can save up to 70% of the resources compared to the naive solution. <s> BIB007 </s> Multimedia Internet of Things: A Comprehensive Survey <s> 1) MULTIMEDIA CLOUD COMPUTING IN IoT <s> With the popularity of multimedia sensing at cloud edges and the reducing cost of the Internet of Things (IoT) fog devices and systems, new challenges have been posed to efficiently deal with the big data multimedia traffic generated from IoT sensing units. Specifically, in this paper we introduce the concept of multimedia sensing as a service (MSaaS), and propose a generalized premium prioritization-based quality of experience paradigm for wireless big-volumes-of-data (BVDs) multimedia communications, with significant energy saving potentials for future multimedia IoT devices and systems. The key contribution of this new framework is its data diversity flexibility at the application layer, which could be flexibly adopted by future multimedia communication systems. Data dependencies in spatial, frequency and temporal domains are analyzed, and interaction with uplink resource allocation optimization are investigated with regards to wireless communication energy cost estimation. Extensive simulation results demonstrate that the proposed prioritization-based communication paradigm has significant energy saving potentials for BVD MSaaS wireless multimedia communications at cloud edges and fogs. <s> BIB008
The ubiquity of audio processing and cameras even in low-end devices increases the audio and visual recognition applications. However, due to small size IoT devices, multimedia data processing requires additional computational services that are readily available. To cope with this issue, the researcher has put forward several remote cloud computing scheme for multimedia processing. Renna et al. in BIB002 presented a framework to provide the optimal balance between energy consumption to process the query and the cost incurred for cloud computing by incorporating resource consumption according to the volume of query which either lies in the idle state or active state. Ubiquitous and heterogeneous data generated from the surveillance system is transmitted to a centralized processing unit over limited bandwidth incur high latency. Ali et al. BIB001 , BIB003 proposed a cloudlet based, lightweight distributed computing framework for IoT video surveillance systems to minimize network latency. The cloudlet system brings cloud computation resources closer to the IoT network to compute critical and sensitive data in real-time. In this article, face recognition is considered as a use case to validate the framework. Cloudlet exists between surveillance cameras and the public cloud. Image/video captured by the camera is sent to the cloudlet for face recognition and feature extraction, which is transmitted to the public cloud for matching and identification purposes. Cloudlet is a device with significant storage, CPU, and GPU. Preprocessing the captured data before transmitting to the public cloud significantly reduces the network latency. DNN based face detection 'OpenFace' in real-time video stream and face denaturing named 'RTFace' technique for privacy and safety has been presented in BIB005 . The proposed algorithm is implemented on cloudlets for speedy and efficient face detection and denaturing. Crowdsensing is an important aspect of smart cities which provides ease in multimedia data aggregation by exploiting multimedia sensor of public mobile phone, wearable devices, and tablets. The acquired data is wirelessly transmitted to the cloud in real-time due to the limited storage capacity of the devices. Hong et al. BIB004 addressed three problems in cloud-based video crowdsensing: 1) optimal transcoding, 2) the optimal data transfer protocol and meta-data selection to upload a video over WiFi, 3) camera parameters which require complete video lookup. The authors presented three algorithms: Adaptive Transcoding Algorithm (ATA) based heuristic algorithm for video representation without quality compromise. Protocol Selection Algorithm (PSA) to dynamically select transmission protocol by comparing average throughput normalized value achieved from FDT and UDT. Cloud Database Algorithm (CDA) based on Fieldof-View (FoV) approach to manage videos in the cloud database. Authors in BIB006 devised an analytical framework to serve the cloud access request from the access network, core, or edge network by maintaining an optimal number of replicas to improve resource utilization. Cloud computing for event detection in a video stream and probabilistic future event prediction is detailed in BIB007 . Authors proposed a dynamic programming based framework to efficiently allocate bandwidth resources from limited uplink bandwidth based on future event predictions to transmit only meaningful information. The results show 80% event detection with 97% QoS satisfaction, utilizing 10% of essential bandwidth. Wang et al. BIB008 devised multimedia sensing as a service (MSaaS), an energyefficient framework work to upload multimedia data by prioritizing it first and then optimally allocate network resources. The authors exploit the quad-tree decomposition algorithm to segregate regular and premium blocks. In this article, the truncation based resource allocation optimization problem is formulated. A unified resource allocation metric is considered based on Automatic Repeat Request (ARQ), channel coding for Forwarding Error Correction (FEC), data rate, transmission power and packet length subject to energy-constrained quality optimization.