Edge computing provides the ability to link distributor users for multimedia content, while retaining the power of significant data storage and access at a centralized computer. Two requirements of significance include: what information show be processed at the edge and how the content should be stored. Answers to these questions require a combination of query-based search, access, and response as well as indexed-based processing, storage, and distribution. A measure of intelligence is not what is known, but is recalled, hence, future edge intelligence must provide recalled information for dynamic response. In this paper, a novel event-oriented indexable and queryable intelligent surveillance (EIQIS) system is introduced leveraging the on-site edge devices to collect the information sensed in format of frames and extracts useful features to enhance situation awareness. The design principles are discussed and a preliminary proof-of-concept prototype is built that validated the feasibility of the proposed idea.
We describe two advanced video analysis techniques, including video-indexed by voice annotations (VIVA) and multi-media indexing and explorer (MINER). VIVA utilizes analyst call-outs (ACOs) in the form of chat messages (voice-to-text) to associate labels with video target tracks, to designate spatial-temporal activity boundaries and to augment video tracking in challenging scenarios. Challenging scenarios include low-resolution sensors, moving targets and target trajectories obscured by natural and man-made clutter. MINER includes: (1) a fusion of graphical track and text data using probabilistic methods; (2) an activity pattern learning framework to support querying an index of activities of interest (AOIs) and targets of interest (TOIs) by movement type and geolocation; and (3) a user interface to support streaming multi-intelligence data processing. We also present an activity pattern learning framework that uses the multi-source associated data as training to index a large archive of full-motion videos (FMV). VIVA and MINER examples are demonstrated for wide aerial overhead imagery over common data sets affording an improvement in tracking from video data alone, leading to 84 detection with modest misdetection false alarm results due to the complexity of the scenario. The novel use of ACOs and chat messages in video tracking paves the way for user interaction, correction and preparation of situation awareness reports. Qualia-based Exploitation of Sensing Technology QuEST is an approach to create a cognitive exoskeleton to improve human-machine decision quality. In this paper, the authors present QuEST-motivated man-machine information fusion with an example for multimedia narratives. User-based situation awareness includes both elements of external sensory perception and internal cognitive explanation. The authors outline QuEST elements and tenets towards a reasoning approach that achieves human intelligence amplification IA in relation to data aggregation from machine artificial intelligence AI. In a use case example for multimedia exploitation, they showcase the need for enhanced understanding of the man mind-body cognition and the machine sensor-based reasoning for establishing a cohesive narrative of situational activities. QuEST tenets of structurally coherent, situated conceptualization, and simulated experience are utilized in organizing multimedia reports of Video Event Segmentation by Text VEST. Surveillance video is used in two key modes, watching for known threats in real-time and searching for events of interest after the fact. Typically, real-time alerting is a localized function, e.g. airport security center receives and reacts to a "perimeter breach alert", while investigations often tend to encompass a large number of geographically distributed cameras like the London bombing, or Washington sniper incidents. Enabling effective search of surveillance video for investigation & preemption, involves indexing the video along multiple dimensions. This paper presents a framework for surveillance search which includes, video parsing, indexing and query mechanisms. It explores video parsing techniques which automatically extract index data from video, indexing which stores data in relational tables, retrieval which uses SQL queries to retrieve events of interest and the software architecture that integrates these technologies.