Humans are rather poor in judging the right speed of video scenes. For example, a soccer match may be sped up so as to last only 80 min without observers noticing it. However, both adults and children seem to have a systematic, though often biased, notion of what should be the right speed of a given video scene. We therefore explored cortical responsiveness to video speed manipulations in search of possible differences between explicit and implicit speed processing. We applied sinusoidal speed modulations to a video clip depicting a naturalistic scene as well as a traditional laboratory visual stimulus , and measured both perceptual sensitivity and cortical responses  to speed modulations. In five observers, we found a clear perceptual sensitivity increase and a moderate SSVEP amplitude increase with increasing speed modulation strength. Cortical responses were also found with weak, undetected speed modulations. These preliminary findings suggest that the cortex responds globally to periodic video speed modulations, even when observers do not notice them. This entrainment mechanism may be the basis of automatic resonance to the rhythms of the external world. The capability of judging the correct speed of a dynamic scene in a video clip is surprisingly poor. We have recently shown that (i) speeding up a soccer match video by as much as 12% goes completely undetected . Thus, iThese findings led us to ask how the speed of a complex video scene is coded in the brain. Taking a somewhat different approach as compared to existing work on visual speed processing ,5, in thFive adult participants aged between 21 and 35  took part in this experiment on a voluntary basis and gave informed consent prior to the beginning of the experiments. They had normal or corrected-to-normal vision and had no history of neurological diseases. This study was conducted in accordance with the principles of the Declaration of Helsinki and the “Comitato Etico San Raffaele”.Observers were seated 57 cm in front of a laptop screen. Visual stimuli  consisteIn a second experimental session, administered on a different day, observers rated in a 9-point scale the perceived strength of the pulsation . Observers were specifically instructed to rate the perceived intensity (strength) of sinusoidal speed modulation. Phenomenally, such rhythmic speed modulation appeared as a pulsation, periodically speeding up and slowing down the stimulus at the experimentally imposed frequency . Stimuli and other conditions were the same as in the first session, except that observers could respond whenever they wished. Overall, the session lasted about 30 min.Scalp electrical activity was recorded through an Enobio device , using the right ear lobe as reference. Electrodes were gel-based passive plates  and were placed on the scalp by means of an EEG cap. EEG traces were band-pass filtered (0.1–100 Hz).To quantify perceptual responses, we used the subjective estimates of video speed modulation strength (pulsation rating). By separating null (rating = 0) and non-null (rating > 0) responses, we transformed gradual judgments into dichotomous yes/no responses, and we applied a signal detection analysis . BasicalTo quantify cortical responses, we used the Eeglab software ERP averaging tool pop_averager  to obtaiBoth the perceptual and the cortical responses were subjected to generalized linear mixed models analysis . The frequency (Freq) and amplitude (Ampl) of video speed modulation were fixed factors, while the video clip (Clip) and participant (Subj) were modeled as random intercept terms to reduce overfitting . FollowiObservers’ capability of detecting video speed modulation is shown in 2 for the sinusoidal fittings ranging from 13% to 36%, which indicated that a relevant component of cortical activity was pulsating at the stimulus frequency. Although an evident oscillation could not be discerned in all traces  were found at all video speed modulation amplitudes, with mean adjusted RIn The present study provided initial evidence for (i) cortical oscillatory responsivity to speed modulation in videos either representing a naturalistic scene (a video clip of ripples on the beach) or RDK , and (ii) dissociation between perceptual and cortical responses, consisting in the presence of cortical responses with the perceptually invisible lowest stimulus strength.In some cases, SSVEPs presented rather smooth, sinusoidal-like signal variations, while in other cases more abrupt changes were observed. In this preliminary study, we took an agnostic position as to the actual shape of the cortical response and used sinusoidal fitting as a simple method to quantify the periodicity of scalp potentials, at the cost of losing the details of the full cortical response. We remark, however, that the choice of using sinusoidal fitting was by no means meant to imply that in our conditions SSVEPs had sinusoidal-like waveforms, as is the case with traditional SSVEP protocols where higher stimulus frequencies are used . Given tA somewhat unexpected finding was an apparent lack of clear regional differentiation in the amplitude of cortical responses. Luminance-evoked VEPs, as well as cortical responses to motion onset ,15,16,17Alternatively, our approach might simply not have been fine-grained enough to detect small regional differences. It is possible that with a more in-depth approach  and a larger sample size, some differences would emerge.We found diffuse cortical responses with the weakest video speed modulation (10%), a condition that was associated with null perceptual sensitivity. This phenomenon calls for subliminal speed entrainment, indicative of an automatic resonance process, which may qualify as a form of attentive subliminal processing , or at lHowever, especially when null results are obtained under low statistical power, it should be borne in mind that “absence of evidence” rather than “evidence of absence” is the proper underlying notion. Furthermore, when the goal is to measure perceptual awareness, null results can be problematic on their own, as it is often difficult to clearly tease apart null from fringe perception . PerceptThe first limitation of this exploratory study is obviously the small sample size. While for some psychophysical designs this numerosity may be reasonable, to provide good statistical power, and also to address possible individual differences, the number of participants should be increased.Another limitation is the small number of EEG electrodes. With eight electrodes, we did not even attempt to compute a spatial map of cortical activity. This issue should be addressed with more sophisticated EEG recordings and analyses. Our goal here was simply to provide initial evidence of cortical entrainment.We should also note a limitation of the experimental design. For practical reasons, we administered the cortical recording session before the perceptual session. This could have introduced a carry-over effect, i.e., observers’ higher sensitivity in the perceptual session as a result of learning. This could be one reason for the steeper rise of perceptual responses with video speed modulation amplitude, as compared to cortical responses. Note, however, that the direction of such carry-over effect, if any , would reinforce the evidence for a dissociation between cortical and perceptual responses: the perceptual responses in the trials with the lowest speed modulation strength were statistically indistinguishable from the responses in the trials without speed modulation, despite the possible beneficial effect of previous exposure to the stimuli.


The INTERPOL protocol must be used in order to identify victims after a terrorist attack.Some adaptations to the abovementioned protocol can be put in place in order to speed up the identification process in such circumstances.The lessons learned from the 2015 Paris terrorist attack can be useful to other disaster victim identification (DVI) units.The terrorist attacks that occurred in France in 2015 and 2016, which had many victims, proved that it is essential to identify victims following the methodology developed by International Criminal Police Organization (INTERPOL) for such events. Initially designed for natural disasters, this approach must be strictly respected should a terrorist attack occur. This includes the specific collection of bodies and body parts, as well as the setup of an antemortem unit, postmortem unit, and Identification Commission. This commission is made up of specialists and will make decisions on the basis of primary identifying elements  and/or secondary identifying elements (other distinctive and particularly discriminating signs). A multidisciplinary team, combining specifically trained police officers and scientists, must provide a reliable and consolidated list of deceased people based on biometric elements cross checked with elements from the investigations. Any list of names generated on another basis should be excluded. Identification of the deceased by relatives , even if the bodies do not appear mutilated and/or decayed, should be avoided to avert erroneous death announcements, body presentations, or even burials or cremations. Similarly, identifying victims only on the basis of their personal effects (such as an identity card) must be absolutely avoided. All bodies, even those whose identities seem obvious to the first responders or to the relatives, must be registered as “X” during the crime scene search and properly identified according to the INTERPOL protocol. The same protocol should be applied to unconscious injured people. Considering the expectations of our modern society for rapid information circulation and quick responses from the authorities, the French team has made a few adjustments to speed up the identification process without compromising its integrity. Validated and supported by both the French judicial and administrative authorities in the light of the experience of the November 2015 attacks in Paris, this innovative method proved its effectiveness during the Nice attack in July 2016. It can only be successful in a context where all the individuals involved in the crisis, up to the highest authorities, understand it, support it, and relay it in the best interest of the victims’ families. During the November 2015 terrorist attacks in Paris, France where 130 people were killed, the urgency of notifying families whether their next of kin was dead or alive led to partial implementation of the disaster victims identification (DVI) protocol. Both the media and the state authorities were demanding an immediate communication of the identities of the victims, with frustrated relatives still waiting to hear news and eager to be taken to their next of kin’s location. Social media did not help: through them, people conducted research by their own means, and performed unverified victim identifications. This “immediacy culture” was associated with a general ignorance and/or misunderstanding of the DVI protocol in France during this national crisis, and had severe consequences for both families of victims and government services. Because of this experience, the various individuals involved in mass murder cases in France have since applied the DVI protocol to all victims, whether they carry identification or not. This was especially the case in Nice in July 2016. The application of the DVI protocol should not prevent the implementation of measures to speed up and adapt this process to ensure both reliable identifications and early notifications to families.Initiated by the Police Unit for DVI  of the Central Department of the Technical and Scientific Police , the “French-style” DVI methodology has been praised in various conferences. These include the annual INTERPOL conference on DVI in Lyon in 2016 and Singapore in 2017, the Australasian DVI conference in 2017, the mass fatality symposium organised by the office of the Chief Medical Examiner of Washington in 2016, and others. Within a scope validated and promoted by the Sous-Direction Anti-Terroriste (SDAT) of the Central Directorate of the Judicial Police , the DVI protocol has been adapted to situations of mass murder and the demands of a society that lives in the immediacy. This methodology is now being used internationally as a reference.Initially designed for natural disaster situations, the DVI methodology is described in a guide first issued by INTERPOL in 1984 and adjusted following the 2004 tsunami in Asia. The aim is to scientifically identify large numbers of deceased individuals using a standardized methodology and a common nomenclature among all INTERPOL member countries. The DVI protocol excludes identification by relatives or by a document bearing the identity of the victim, such as an identity or transport card. Specialized units are specifically trained.The “antemortem unit” collects information regarding the physical elements of the missing person  and other technical elements (possible papillary and genetic traces) from families . Any information collected by another unit relating to the missing person should be transmitted to the antemortem unit.The “postmortem unit” simultaneously records all materials found on the deceased victim that could be used for their identification . They are surveyed for papillary traces and DNA sampling. This unit, together with the relevant specialists, lists all physical, medical, and dental characteristics of the body. Any elements that may possibly assist with identification of the body are photographed. The postmortem unit is therefore usually located inside the morgue, or in any other improvised location for gathering bodies if there is no morgue available.The Identification Commission is made up of representatives of the ante- and postmortem units, and at minimum one forensic physician, one fingerprints specialist, one DNA expert, one odontologist, and if possible an anthropologist, police investigators in charge of the case, and the Public Prosecutor’s representative. This commission is held once the ante- and postmortem units have collected their information and after a comparative synthesis (reconciliation) has been performed. The Identification Commission declares formal identification of the individual if the data collected from the body postmortem match those collected by the antemortem unit.The list of identifications validated by the commission shall be transmitted without delay to Public Prosecutor. The notification of death can be made by an investigator and the DVI staff that followed the family on the antemortem side, and the body may be presented if the family so wishes.Because the DVI Police Unit works on demand and under the scope of the Investigation Service , it works within judicial proceedings. The antemortem and postmortem files are attached to the identification minutes generated during the commission.The DVI methodology has been carefully developed and tested at an international level, within the framework of an INTERPOL working group that has helped it evolve for over 30 years from the experiences of each member country. France is part of this working group, and our counterparts are scrupulously applying this methodology in the case of a terrorist attack .Even in a disaster situation on national territory, such as that of November 2015, there are often many foreign victims . Our foreign counterparts therefore expect France to identify their nationals according to the established standards. It should be noted that the DVI teams from these other countries follow the INTERPOL protocol by preparing the antemortem files, which they later send to the French DVI unit for reconciliation with the postmortem data they collected.Even if the bodies are in good condition (face not deteriorated), the large number of them inevitably creates confusion at the scene of the attack. Personal effects can get mixed up. In the following days, many government services are involved and broadly requested to account for the victims. Coordinating all of these individuals is complex; if the procedure and protocol for the list of deceased victims are not rigorously flagged and standardized, then incorrect lists can circulate, and families can be given inaccurate information.ab initio. This makes the task even more complicated because some of the missing people may be unconscious in the hospital or even safe elsewhere. A very large number of families are therefore looking for their loved ones very early on following the incident.Unlike in an aircraft crash situation, for example, no list of missing persons is available Even if they remain unscathed, the faces of the victims are transformed by death. Following a major traumatic injury, families are not always able to recognize their loved ones. Some people might be mistaken in their recognition of victims, as was observed in November 2015 in Paris.“Was the child I recognized a year ago my own?”This question may come to haunt families years later. Because scientific, simple, fast, and safe identification methods are available, the responsibility for identification must not solely rest on the victims’ families.ab initio (those who carried identification such as an ID card and were thus presumed to be identified outside of the DVI protocol), problematic cases from the DVI perspective have been identified: identity swaps, incorrect death notifications, reverse presentation of bodies … namely more than 10% of the victims. The 40 victims not carrying identification documents, for whom the DVI protocol was applied, were identified with no error. In the November 2015 case, the 90 victims that were not identified through the DVI protocol, but rather presumed identified and directly presented to their next of kin, were still processed by forensic pathologists. The pathologists recorded important information such as DNA. Therefore, should any future doubt arise among the relatives regarding the individual’s identification, antemortem research could still be performed, and lead to a scientific confirmation of the victim’s identity.In November 2015, among the 90 victims considered to be identified Scientific identification of all victims, even those carrying identification documents, is essential to avoid inaccurate early death notifications or presentation of bodies, as well as even incorrect burials or cremations. These events would result in major distress for the relatives, and possibly significantly prolong the search for their loved ones.From November 2015, the DCPJ, together with the Interministerial Cell for Helping Victims , the Counter-terrorism section of the Paris Public Prosecutor’s Office , and the largest French morgues , have developed a mechanism to speed up the scientific identification process without compromising the integrity of the results. The “French style” protocol, now set up in interministerial guidelines [Dealing with victims of terrorist acts”, such that the collection of information on missing persons and death notifications took place in a much more serene and comprehensible way for families following the attack in Nice in July 2016.Numerous interministerial meetings and in-depth exchanges between the DVI Police Unit and its partners  have enabled the various individuals interacting with the families of victims to understand the scope of each other’s missions and to coordinate together. This led to a clear procedure for families and a rewriting of the interministerial directive, “To avoid swapping and confusing identities, no identity should be affixed to a victim when an inventory of the bodies is generated on the attack scene. The INTERPOL protocol advocates the following: each body must be singled out by a postmortem number; all victims must be registered under X , even those carrying ID. A simplified registration form has been distributed in all judicial police departments in France, as well as in the Paris Police Prefecture. This single sheet, in French, summarizes the essential information to be recorded when collecting a body, the manner in which it must be done to secure the later DVI process, and how to attribute a postmortem number. The form is an integral part of the investigators’ methodology on an attack scene and allows for the correct listing of bodies during major crime scene findings, even if the officers involved in the collection are not DVI trained. Thus, the DVI Police Unit can efficiently take over the process when they arrive at the scene.The search for body identifiers carried out by the DVI postmortem teams is less technical and faster than an operation for determining the cause of death performed by a forensic physician (an autopsy or a less comprehensive examination). These postmortem operations can be performed one after the other.In November 2015, the identification-related sampling was carried out simultaneously with the search for causes of death. Postmortem teams thus depended on the pace of autopsies. Given this experience, the French DVI police unit decided to separate these two actions and to carry out the identification operations first, which can considerably reduce the identification process time.It is not necessary to wait until all identification operations are completed to begin the comparisons between postmortem and antemortem elements within the Identification Commission and validation of the identified persons to fill the sole list of victims and make the death notifications. This meeting can be held daily, allowing families to be notified as early as the first days following the incident.The abovementioned simplified record sheet indicates whether an identification document, telephone, or any other evidence suggesting the individual’s identity is found on the victim’s body. When it is the case, it is possible to carry out the postmortem operations on the bodies of these victims first (before the other bodies), and to simultaneously collect antemortem information from relatives whose identity seems linked to the victim, to quickly and scientifically validate or invalidate a suspected identity. This validation can happen as early as during the first identification commissions.Tattoos, scars, personal effects, and piercings are all considered “secondary” elements, which, if the bodies are not fragmented, can enable quick identification without having to wait for primary identifiers  antemortem that may be hard to obtain. If these secondary elements are strong and/or numerous, then it is possible to provide an identification during commission on this basis, as provided in the INTERPOL guidelines . HoweverPreviously largely ignored by the teams of the various morgues in France, the DVI protocol was widely disseminated after the November 2015 attacks to the largest morgues in France, including those in Paris, Bordeaux, Lyon, Strasbourg, Marseille, and Lille.During these exchanges between the DVI Police Unit and morgue teams, it was indicated that a dedicated space within the morgue was needed for the postmortem identification phase. The configuration of each morgue was studied to incorporate this requirement. In some morgues, detailed documents were developed, including a model action plan. An intervention by the DVI Police Unit at the Legal Medicine Congress in 2017 also made it possible for these recommendations to be widely disseminated to all head pathologists at morgues in France.ad hoc morgue should be secured so that only families duly authorized by the DVI Police Unit, following identification by the Identification Commission, can access it. An appointment is set by the DVI Police Unit and the morgue if the families wish for a presentation of the body following the notification of death at the Family Reception Center.Finally, all morgues have needed to turn away families who came on their own to seek out loved ones after a terrorist attack. The The morgue forbids any initiative/uncoordinated communication between the DVI Police Unit and families during the time of the crisis and redirects family phone calls to the antemortem unit.To test these new modes of operation, interdisciplinary exercises have been performed on numerous occasions in various places in France, which allowed for the validation of the new protocol’s effectiveness and the articulation and immediate recovery of information at the Counter-Terrorism Prosecutor’s Office, CIAV, and within morgues. Additionally, those exercises validated the effectiveness of the separation between the identification process on the one hand, and the process of looking for causes of death on the other hand.Only the Counter-Terrorism Prosecutor’s Office is permitted to communicate publicly about the number of victims and their identities. It does so only after the identification commission to avoid spreading inaccurate information. Updates to victim reports, also nominative, are only published by the Prosecutor’s Office once the families have been notified by the DVI Police Unit, not immediately following the identification commission or by the press to maintain privacy and respect for the families.The need to apply the protocol does not stem from one individual’s state of clinical death, but because they cannot directly provide their identity. This is one of the most significant developments of the doctrine. In the chaos of mass murder, the traceability of a victim's identity is difficult to ensure. Families looking for loved ones are not aware if they have died or been hospitalized. Their loved ones could be completely unconscious in intensive care in a hospital, making the response to families all the more urgent. Identifying information should be collected from both those who died and those who are unconscious in the hospitals. Faced with the sometimes-significant injuries of unconscious people who are being treated, including head injuries, it is essential that hospital staff do not make hasty and potentially inaccurate identifications of their patients .In the case of a terrorist attack, the DVI INTERPOL protocol must be adhered to in the interest of families. No death notifications or body presentations should be made before validation by the Identification Commission. However, this protocol must be implemented in an efficient and quick manner to reduce the families’ unbearable waiting as much as possible. This cannot be achieved without support from the highest authorities of the State. A clear and consistent way of addressing the families is necessary. They must understand that the specialized services are making every effort to provide them with a consolidated response as soon as possible.Through antemortem interviews and the elements they provide, the families of victims often feel they are actively involved in identifying their missing loved ones and therefore concur with the DVI’s approach. This can only be done if the State is organized to implement this in a coordinated and comprehensible manner, as it did during the Nice attacks in 2016. This was done in light of its response to the November 2015 attacks in Paris. Thus, 88 victims in Nice were identified in only 4 days without error  in keeping with the DVI protocol and without any visual recognition by families. The protocol was also applied during the terrorist attacks in Trèbes (March 2018) and Strasbourg (December 2018).As the repeated feedback shows , our foreign counterparts are meticulously applying this doctrine. The process is welcomed by the families of the victims themselves, most recently in Belgium, after their 2016 terrorist attacks. France will not remain on the outskirts of this global movement. By systematizing its application, the French DVI Police Unit has been able to make innovative adaptations to the identification protocol in the event of a terrorist attack. They also demonstrated its relevance, operational nature, and benefits to families and society in general.


In this paper, we present HOMER, a cloud-based system for video highlight generation which enables the automated, relevant, and flexible segmentation of videos. Our system outperforms state-of-the-art solutions by fusing internal video content-based features with the user’s emotion data. While current research mainly focuses on creating video summaries without the use of affective data, our solution achieves the subjective task of detecting highlights by leveraging human emotions. In two separate experiments, including videos filmed with a dual camera setup, and home videos randomly picked from Microsoft’s Video Titles in the Wild (VTW) dataset, HOMER demonstrates an improvement of up to  Several years ago, the limits of digital storage compelled people to only keep their best photos and video highlights. Today, the improvement of digital storage capacity, the emergence of the cloud, the advancements in compression standards, and the prevalence of social networks have allowed users to increase their generation and consumption of pictures and videos. According to Cisco, videos are responsible for 75% of global IP traffic, and this proportion is anticipated to increase to 82% in 2022 . The larThe focus of this work is to automate the generation of video highlights, thus saving time for people manually developing their own montages and scanning through hundreds of media files to find video highlights. Besides home users, automatic video highlight generation can also lead to a wide range of other applications such as interactive browsing ,4, searcFacebook, Apple and Google have developed similar technology, with limited success. Facebook’s “One year in review”, a feature summarizing the user’s year based only on uploaded photos and videos, considerably constrains the available content. This automatic and unsupervised generative feature also raised ethical questions due to the Al-Qaeda scandal , where “In the field of video summarization, external webcams are utilized to detect user’s facial expressions ,10,11, eAs opposed to video summarization, where the aim is to summarize an entire video, this work focuses on extracting video highlights based on affective content, which might not be present in emotionally neutral videos where no stimulating event occurs. Wang et al.  describeThis work addresses the lack of high-quality video highlights generation with HOMER , and the relevance of audio  ,7,27,28.Early research extracted audio-visual features and used self-designed algorithms to transform them into segment boundaries, often derived from a resulting score function . The selAlthough most of the works such as  use exteHybrid content techniques must deal with the constraints of external content sensing and the fusion of external and internal information. A hybrid affective approach was driven by Peng et al. , who proState-of-the-art solutions are a significant improvement in the field of internal video summarisation, largely due to deep learning breakthroughs. However, the absence of user emotion perception makes the existing models well-adapted for video summarization, but irrelevant for the extraction of highlights. The high subjectivity of the highlight semantic makes external content-based techniques more suitable for the task. The ideal case of hybrid solutions is rarely visited, pushing us to develop a hybrid solution.In summary, we identified four types of limitations present in current literature in automated highlight generation. The first limitation is the scope of videos analyzed; focus on a specific type of video, such as sports or video games, limits the generalizability of these works to a specific setting, and may not work well for videos outside of that setting. The second limitation is a reliance on audio or visual features; relying solely on the video content can reduce the personalization of highlights which can be different between people. The third limitation is the usage of external hardware, which can be cumbersome to setup or requires human interaction with the system. Finally, this is the first system developed on a mobile platform, which is not only a popular medium for capturing video, but also has the potential for more widespread dual camera video capture in future generations of mobile technology. The limitations of relevant existing works in highlight generation are shown in In this section, an overview of the overall system architecture is presented, followed by a detailed stage-wise analysis of each component.Due to a combination of computational requirements for video processing and our desire to create a web service that can enable multiple third party applications, HOMER is currently implemented on a cloud server rather than a mobile device. As there is no real-time requirement for our applications, a cloud server allows for significant energy savings, as well as flexibility in development for future improvements. A major concern of cloud-based solutions is data privacy, especially as sensitive data are sent to the server. We address this concern in HOMER by storing data securely during highlight extraction, and deleting all data immediately after. In future works, pre-processing (such as obfuscation or feature extraction) of sensitive data can be performed on the mobile device to eliminate the need to send sensitive data to the cloud.As shown in Scene Video: video to be highlight-extracted. Both video frames and audio signals are extracted as distinct raw signal inputs for the highlight extraction pipeline.Facial Video: video of the user’s face recorded in reaction to the recorded scene. Its duration must be equal to the scene video duration.The input stream provided by the client is composed of the following signals and parameters:The three input signals  should be pre-processed before feature extraction to generate homogeneous data that can be fused to make highlight decisions.As shown in previous works ,42, a vi(a) Face Detection.In some cases, the user’s full face may not be visible in the frame. HOMER utilizes face detection to improve the emotion recognition model through data standardization. Using the Haar-cascade open-source face detection algorithm, the user’s face is cropped from the raw image in each frame a. In cashttps://github.com/piyush2896/Facial-Expression-Recognition-Challenge) to the Facial Expression Recognition Challenge on Kaggle. The model achieves a desirable compromise between computational time and performance, with an accuracy of 69% over seven classes  on the AffectNet dataset [(b) Emotion Recognition. Most state-of-the-art models achieve emotion recognition through two different approaches: classifying extracted action units (AUs) of the human face ,51 or pe dataset . As show(c) Peak Detection. From the two resulting signals (happiness and surprise), specific event features had to be extracted. A peak detection algorithm was developed to extract peak sides and peak top c. As the(d) Feature Extraction. Each emotion peak represents an event and is described by five features: the start of peak rise, the start of peak top, the end of peak top, the end of peak fall and the average value of peak top. The resulting simplified signal from this feature extraction is shown in HOMER utilizes the frame similarity (FS) between consecutive frames of video to improve highlight detection robustness. Frame similarity was used in order to detect dynamic state changes throughout the recording, such as differentiating static camera and camera pan. This feature improves the precision of the highlight boundary decisions by enforcing the boundaries to correspond to the start and end of a camera dynamic mode. For instance, if a camera pan occurs at some point in the video, the start and end of the pan will be prioritized over other timestamps. Frame similarity also helps to increase highlight content diversity by prioritizing instances where the camera moves between two scenes, resulting in a highlight based on the first scene, content between both scenes, and the second scene. An example of the corresponding frame similarity pattern is shown in To detect these dynamic changes as one-time point features, the following steps were performed:(a) Frame Extraction. The frames are extracted from the video with a higher rate than emotion recognition due to the fast possible variations of the signal a.(b) Frame Similarity Computation. Frame similarity was computed pixel-wise using the work done by Zhang et al. . This me(c) Discontinuity Removal. As shown in (d) Low-pass Filter. The signal is smoothed using a low-pass filter with a cutoff frequency of 2.5Hz. The resulting signal provides an alternated succession of local maxima and minima.K is the total number of frames. We identified all the local maxima of S as follows:(e) Feature Extraction. We define the pre-processed frame similarity function as:F of feature points, only the local maxima subject to high variations were kept with the following condition:To build the final set Each feature point is associated with a score proportional to both the total amount of local variation and the difference between right and left variations. Finally, we observed that the main patterns of similarity have varying importance. The score is re-scaled accordingly to these particular categories using fuzzy logic.“Hills”: these frame similarity peaks translate either to brief camera stop-motions or noise, and thus are assigned low importance.“Plateau”: this pattern translates to an extended camera stop, often due to an event of interest. This frame similarity shape also appears when a camera zoom-in/zoom-out is performed. The medium level of importance was chosen instead of high because of the lack of summarisation this pattern offers. Only one scene of the video would be contained in the generated highlight, versus two for the “Valley” pattern (see below).“Valley”: this pattern often translates to a camera pan or tilt, from one motion stop to another. This pattern was assigned high importance for its semantic generalization. This pattern contains information from the first motion stop scene, information from the second motion stop scene and information between both scenes. Furthermore, the in-between content often presents important information, due to the constant low similarity value. As an example, this pattern occurs when the user is performing a constant pan/tilt to show an area or a beautiful landscape.The three commonly encountered patterns are :“Hills”:Each frame similarity feature point is now fully described by two parameters: its frame number The sound of a video is a meaningful indicator of events and can deliver additional information about the video semantics and affective content. Many state-of-the-art techniques combine event detection and event recognition into deep learning models, sorting out polyphonic sound event classes ,56,57. T(a) Event Detection: The sound signal of a video can be segmented into different events based on amplitude. The higher the sound energy is, the more likely that the content will be important or affective. To detect and segment high-energy events relative to the rest of the signal, we pre-processed the signal accordingly and designed a bottom-up unsupervised event extraction .Pre-processing: Firstly, sound samples which only contain high amplitude noise are discarded. To detect such cases, the energy density of the whole sound sample is computed. This metric is independent from the sample size:W above the threshold Any sound sample having After normalization, we use the wavelet transform to transform the signal Event extraction: Sound event extraction is performed using an unsupervised bottom-up approach. After pre-processing, the sound wave is transformed into a succession of peak-shaped sub-events. To extract events, relevant sub-events are clustered into meaningful high-level events using a threshold Algorithm 1 Adaptive threshold computationRequire: sound signal of n valuesEnsure: adaptive threshold value 1:2:3:4:ifthen5:return 0 6:end if7:if elbow present in then8:return 9:end if10:ifthen11:return 12:else13:return lowest  14:end ifd signal b, resultThe sub-events are subsequently clustered into high level events by Algorithm 2. The algorithm takes three primary inputs: the maximal time gap allowed one to merge two sub-events  progressively merges sub-events by comparing sub-event b-events b. Once aAlgorithm 2 Sound events clusteringRequire: sub_events D, Ensure: events E1:y = amplitude of a sub-event/event2:X ← first cluster to be grown, initialized with 3:for each sub_event do4:if (then 5:  X ← X + 6:  y ← new weighted amplitude of X7:else 8:  insert 9:end if 10:end for11:for each pair (do12: 13:if (then 14:  Merge 15:end if 16:end forThe second loop (lines 11–16) considers every two consecutive cluster events. The two clusters merge into a bigger event if they meet the time distance condition and either meet the score distance condition or if the event with lower score has a duration below the threshold b-events  by incre(b) Event Recognition: To provide more information on the segmented events, a sound event classifier was used to label each event with one of three labels: speech, laughter, or miscellaneous . Speech A speech versus non-speech binary classifier and a laughter versus non-laughter binary classifier were combined to form the three-class classifier in a one-versus-all strategy. Each binary classifier includes two parts, adapted from Wagner et al. : first, This approach was chosen over conventional feature sets such as the mel frequency cepstral coefficients (MFCCs), because the feature representation learned by the convolutional network demonstrated greater robustness with real-world noisy audio data, corroborating work done by Choi et al. .(c) Segmentation and Recognition Fusion: Event detection and event recognition are combined under the assumption that a detected event is homogeneous, containing one class only. Based on majority vote, a segmented event is labelled with the predominant class, as shown in In existing research, fusion of multi-modal features is performed on non-time dependant features extracted from time-dependant signals. These approaches do not require synchronization between features, and are typically thrown into classifiers, such as support vector machines (SVM) ,61, BayeM is the total number of features derived from sound, frame similarity and emotion and N the number of timestamps  to facilitate the synchronization of the three input signals . As presented in  signals  is projeThe event timeline is used to decide the start and end times for highlights. We implemented two different approaches in HOMER: a machine learning approach and a hand-designed approach.We present the model design, analyze the results obtained, and discuss the limitations of machine learning in this application.(a) Model Design: A machine learning solution is a convenient option, as a deep understanding of the causality between features and output is not required. Especially in the case of highlight extraction using multi-modal inputs, constructing a hand-designed model is a non-trivial task. The main challenge is to manually annotate video highlights to supervise the model training. Each video may contain one or more highlights, labelled with start and end timestamps. A sliding window of 5 s is used to build fixed-size inputs from the ET, each containing exactly one ground truth label (either highlight start or end).In contrast to previous works, where video highlight/summaries are generated with a single score function representing frame-level importance ,15,18,19The classification is multi-class (20 classes), each class being a time-stamp within the window input but with a lower fps. Fewer output classes than input points allows for easier learning and decreases the required model capacity. During testing, each classifier outputs a start/end probability score for each class, resulting in two score functions .(b) Classification Results: Each classifier was trained using 2000 input samples derived from 300 labelled videos. Different models were used initially to determine whether machine learning was an appropriate approach: logistic regression (LR), support vector machine (SVM), k-nearest neighbors (kNN), random forest (RF) and adaboost. The results of the highlight start classifiers are presented in The results are globally poor and perform slightly better than a dummy classifier.(c) Machine Learning vs. Hand Designed Algorithm: Supervised machine learning models are able to learn complex nonlinear problems, with the only requirement being sufficient training data and their associated ground truth labels. To the human operator, a deep understanding of the input/output causality is not required, as long as the causality exists and the operator is able to derive the output of a given input. Nevertheless, the function learned by the model, which tends to describe the reality as faithfully as possible, is a series of deterministic computations. The ground truth labels of the classifier, start/end of highlights, were manually set by looking at the original scene video. Given that highlight labelling is a subjective task, the labels significantly differed from one subject to another. This low input/output causality led to the poor performance of the machine learning classifiers.Similar to the machine learning model, the empirical hand-designed algorithm extracts highlights by computing two start and end score functions respectively called  account a, specif(a) tion see b: in thetion see d is a liWe define the set of sound events Intuitively, the score functions are set to low values while the sound event is occurring and are set to high values after the event’s end. The rationale is that highlights should not be cut during a Equation .The frame similarity function Equation ) was desThe function pattern associated with an emotional event is represented in (b) ion: see c: ConsidThe start function . This pEquation , by inteN.B.: because of the variable change t’= Equation , increasWe chose the integration of sound events to promote successive close events in the timeline. Close events usually belong to the same semantic scene and integrating allows the combination of the whole semantic scene. The score function would then linearly grow for each new event, pushing s formed .The pattern used for the emotional events is represented by the teal line in α and tβ d.As explained in –The score of –Another input parameter explained in The output of the highlight extraction algorithm is either The timestamps HOMER supports a large range of new applications, as demonstrated through the implementation of two novel applications on an Android mobile device.A primary application of HOMER is the use of both the front and back camera of a mobile device simultaneously . When a Both videos are sent to the web service and fed as inputs to the highlight generation algorithm. Once generated, the highlight(s) are sent back and stored locally on the phone for later browsing in the Android application. A user interface  allows tThe second application, also implemented as an Android application, aims to use the highlight generation service to create memorial video albums . SimilarIn order to take advantage of HOMER’s functionalities, the user is asked to choose a time range corresponding to the start and end times of the memory period . The vidEven though applications 1 and 2 harvest the scene video in two different ways , the videos are handled in the same way, and are sent with the corresponding facial video to the server. When a user rewatches a video, the user is still predisposed to show emotions.We outline two other applications that can leverage the highlight generation service.With the emergence of the streaming platform Twitch, watching gamers play video games live is a rapidly growing source of entertainment . Often, per week , and curVideo calling applications such as FaceTime and Skype canonically involve two facial videos, but one can be treated as the scene video as well. In that case, highlights of a video call can be generated, for example capturing funny jokes or exciting parts of the conversation.As explained in To validate HOMER’s performance, we conducted an experiment to validate HOMER’s performance against baseline algorithms.For evaluation, we uploaded the automated highlight generation application on an HTC One M8 smartphone, which supports the simultaneous recording of front and back cameras. While this capability is not enabled in some versions of Android and iOS, it is not a limitation of the smartphone hardware, and can be activated by software updates.In this experiment, a set of 20 videos was randomly chosen from the Video Titles in the Wild (VTW) dataset supported by Microsoft Research . The setA set of 20 videos was randomly chosen from the Video Titles in the Wild (VTW) dataset supported by Microsoft Research . The datMoreover, 10 subjects were recruited for the experiments to promote reliable statistics and human diversity, since emotional responses to a video can vary significantly from person to person. Subjects were aged between 19 and 30, three female and seven male, with varying educational backgrounds (from arts to engineering). Subjects were recruited with IRB approval. In the first experiment, after recording a video from the front and back cameras, subjects were asked to label the video highlights by start and end time. In the second experiment, subjects were asked to watch the set of VTW videos in a fixed randomized order on a mobile device, with their facial reactions being recorded and the video highlights labeled. In addition, subjects were shown the output of HOMER and asked to rate their satisfaction with the generated highlights as “satisfied”, “partially satisfied”, or “not satisfied”.The generated highlights were evaluated frame-wise using The Pronounce level model presented by Joho et al. , which rIn addition to the Pronounce level model, we also implemented variants of HOMER which only utilize frame similarity or audio extraction for highlight extraction. For these two baselines, the implementations are identical to HOMER, except for the inclusion of only a single input source. For example, the frame similarity pipeline calculates the begin/end of highlights as quations  and 4)..fα(t)=fFTo evaluate highlight generation, 20 videos were taken from the VTW dataset. The video duration ranged from In the evaluation experiment, subjects annotated Finally, we present the cumulative satisfaction ratings for the generated highlights in Intuitively, the priority is that the generated highlight is satisfactory to the user, as opposed to precisely capturing exactly the highlight that the user has in mind. For example, some users might prefer more context, desiring additional frames to precede the true “highlight event”, but they are satisfied as long as the “highlight event” is captured. Thus, As mentioned in Other works, such as ,46, requMachine learning methods for internal highlight generation have been widely explored and have shown good results. In this work, we explored using deep learning to generate start and end timestamps for highlights, as in We present HOMER, a cloud-based system for video highlight generation. Our hybrid approach achieves high performances and outperforms state-of-the-art solutions by fusing internal video content-based features with user’s emotion data. Our perception-based solution makes it possible to detect user relevant video highlights. In two separate experiments, including videos filmed with a dual camera setup, and home videos randomly picked from Microsoft’s VTW dataset, HOMER demonstrates an improvement of up to


It results from the equilibrium of adenylate kinase (AK), which uses Mg-chelated and Mg-free adenylates as substrates in both directions of its reaction. The AK-mediated primary control of intracellular [Mg2+] is finely interwoven with the operation of membrane-bound adenylate- and Mg2+-translocators, which in a given compartment control the supply of free adenylates and Mg2+ for the AK-mediated equilibration. As a result, [Mg2+] itself varies both between and within the compartments, depending on their energetic status and environmental clues. Other key nucleotide-utilizing/producing enzymes  may also be involved in fine-tuning of the intracellular [Mg2+]. Changes in [Mg2+] regulate activities of myriads of Mg-utilizing/requiring enzymes, affecting metabolism under both normal and stress conditions, and impacting photosynthetic performance, respiration, phloem loading and other processes. In compartments controlled by AK equilibrium , the intracellular [Mg2+] can be calculated from total adenylate contents, based on the dependence of the apparent equilibrium constant of AK on [Mg2+]. Magnesium signaling, reflecting cellular adenylate status, is likely widespread in all eukaryotic and prokaryotic organisms, due simply to the omnipresent nature of AK and to its involvement in adenylate equilibration.Free magnesium (Mg Magnesium (Mg) is one of the most abundant cations in living cells, second only to potassium . Total M2+) is in the order of 1–5 mM [2+] may decrease well below 1 mM, e.g., to 0.2 mM in chloroplasts during the induction phase of photosynthesis (see below). Under these conditions, the proportion of Mg-chelated NTPs decreases. The same rule applies to ADP and other nucleoside diphosphates (NDPs), and to a certain extent to AMP and other nucleoside monophosphates (NMPs).Chelation of nucleotides by Mg is an essential feature of cell metabolism. Among nucleotides, adenylates  are the most abundant, with ATP being produced both during the reductive (photosynthetic light reactions) and oxidative (respiration) phosphorylation . Most, if 1–5 mM , 95–99% f 1–5 mM . In someKMgATP) of 73 mM−1, which implies that a 1:1 (molar ratio) mixture of Mg and ATP will result in nearly all of Mg and ATP tied up as MgATP. The binding of ADP (or other NDPs) by Mg is less strong, with stability constant (KMgADP) of 4 mM−1. This implies that, in physiological conditions, roughly 30% (or less) of ADP exists in the cell as free ADP, and 70% (or more) as a complex with Mg (MgADP) (i), another high energy storing compound, is also highly dependent on [Mg2+]  A 15]. T. TKMgATPn [Mg2+] B.2+], is set as a signal from total adenylate pool in a given cellular compartment. The [Mg2+], in turn, depends on metabolic status of the cell, and on rates of transport of Mg2+ and adenylates on specific transporters (see below). AK in different cell compartments establishes the concentration of [Mg2+] via equilibration of adenylate species, as well as membrane potential of cell organelles [2+] in a controlled way depending on the rates of ATP production and consumption and, in turn, optimizing these rates. This [Mg2+] directly regulates multiple enzymes and translocators, thus representing a powerful feedback signal from the energy level of the cell and its compartments expressed in the concentrations of adenylate species.Both Mg-complexed and free adenylates are equilibrated by adenylate kinase (AK), a ubiquitous enzyme present in all organisms. The result of AK-maintained equilibrium, [AMP]/[ADP]2, depends on free magnesium (Mg2+) concentration in the reaction mixture and can be described as a bell-shaped curve, peaking at 1.5 at ca. 0.2 mM Mg2+ Kapp of AK can be calculated from the experimentally determined total of each of the adenylates taking part in the reaction  [AMP]/[MgADP][ADP], is not dependent on [Mg2+] and has a fixed value of ca. 5.5  . In cont ca. 5.5 . The nonfied AKs ,54,56,57fied AKs ,49.Kapp of AK, although different from the Ktrue, can be a useful parameter linking concentrations of all adenylates to [Mg2+]. Thus, knowing the contents of total ATP, ADP, and AMP in a given biological preparation, and assuming that they are under equilibrium governed by AK, can provide information about intracellular [Mg2+]. Similarly, when only [Mg2+] is known, this can be linked with a given Kapp of AK  via the corresponding buffering equilibria of NDPK and UMP/CMP kinase. The latter reacts reversibly with UMP and CMP rather than AMP, and uses ATP as a second substrate . With respect to UMP/CMP kinase, the Kapp of its reaction (UMP/CMP + ATP ↔ UDP/CDP + ADP) is likely to depend on [Mg2+] in the same way as AK. Additionally, regardless of what true substrates are for NDPK and the other kinase, cellular pools of non-adenylate nucleotides are generally much smaller than those of adenylates [2+] would still respond more strongly to AK-mediated equilibrium than that of NDPK and UMP/CMK kinase.When considering the Mg requirement for NDPK, its true reaction can be described either as MgATP + NDP ↔ ADP + MgNTP, or MgATP + MgNDP ↔ MgADP + MgNTP. To the best of our knowledge, it is unknown whether nucleoside diphosphates (including ADP) used by NDPK are reactive as Mg-chelated or Mg-free species. This could, at least theoretically, make significant difference in terms of the relationship between the enylates . Thus, iKapp) of AK is very much dependent on [Mg2+]  theory, developed and popularized by Atkinson . The then [Mg2+] , and is lability . These alability ,59 and Plability , and we 2+. It has been reported that up to 90% of the cytosolic pool of nucleotides is bound to Mg , because of the lack of an Mg2+ transporter in their membrane, but they also constitute only a tiny fraction of cell volume, thus are unlikely to contribute significantly to overall cellular Mg2+ homeostasis. Both ER and peroxisomes do not have their own AK, and adenylate metabolism there must be independent of AK equilibrium.To the best of our knowledge, there are no data on  in plants most frequently have been performed with Mg-binding ionophores, using fluorescence spectrophotometry to detect the Mg–ionophore complex, sometimes also with help of fluorescence microscopy  in whole cells, the cytosol, and organelles  values that were derived from adenylate contents are comparable to those obtained by other methods , cytosol, nucleus and the IMS of mitochondria . The oumination .Kapp of AK for such a system will be a mean Kapp of all cellular AKs and should apply only to compartments where AK equilibrium is established. This excludes peroxisomes, ER, and vacuoles, which lack AK isozymes , whereas the pool of adenylates in the ER might be significant, given that the ER represents a continuous membrane system, often quite abundant in the cytosol.A different situation occurs if the adenylate data are collected for whole tissues, cells, or protoplasts, where mixing up of various adenylate pools occurs, and the final result reflects the adenylate status in whole tissue/cells, but not in given compartments. The calculated isozymes . Among tKapp of AK to derive [Mg2+] from total contents of adenylates in tissues or whole cells, the main drawback of this method is that it yields an average [Mg2+] for all compartments where AK equilibrium applies . This approach is obviously only an approximation , but it could be useful, especially when studying a process known to be confined to a single compartment, where substantial changes in [Mg2+] have already been observed by other methods, e.g., during the light-induction phase of CO2 fixation in chloroplasts  than old ones), light conditions (darkened leaves have lower [Mg2+] than in illuminated ones), and salt stress conditions (stressed plants have lower [Mg2+]). For instance, based on data from Nieman et al. [2+] in young and mature pepper leaves was 3 and 6 mM, respectively, whereas salt stress decreased [Mg2+] in safflower buds (from 10 to 4 mM). When applying the same approach to adenylate data from one study on the effects of salt (NaCl) stress on energetics of cyanobacteria [2+]. When stressed, the cyanobacteria maintained low [Mg2+] levels of 0.24 mM, which markedly increased to 2.2 mM upon salt withdrawal. This probably reflects the observed two-fold higher photosynthesis rate of these microorganisms under normal conditions, resulting in more ATP produced and thus requiring more magnesium. Cyanobacteria are prokaryotes and do not have any organelles; therefore, aside from thylakoid-like membrane system, the calculated [Mg2+] might be representative of [Mg2+] anywhere inside of their cells.In earlier studies , based on et al. , [Mg2+] bacteria , the cal2+ traffic into and within a cell. Mg2+ is taken up first by specific translocators in the plasma membrane, and then it is distributed to several membrane-surrounded compartments, each using its own specific set of Mg2+ transporters  exchanger superfamily  in regulating a supply of free adenylates to the translocators.In Arabidopsis, at least 16 distinct genes for adenylate carriers have been identified , which cent, ADP . It is bchangers , use frechangers ,100, impAdenylate translocators have variable organ/tissue-specific expression patterns under different environmental conditions and at different developmental stages, suggesting specific non-redundant functions for each of the translocators . For ins2+, and the chelated and free nucleotides are frequently key substrates/effectors in metabolism. The same concerns pyrophosphate (PPi), an alternative energy currency, which is active as an Mg-chelated or Mg-free species; (ii) Binding of Mg2+ frequently modulates and stabilizes activities of enzymatic proteins involved in cell energetics processes, but also in DNA replication, transcription and translation and other processes; (iii) The Mg-chelated and free adenylates govern various aspects of cell energetics, such as rates of energy metabolism, translocation of adenylates across membranes, or contribute to allosteric regulation of metabolism; (iv) The substantial changes in intracellular [Mg2+], as in the cytosol of cells under anoxia, may reflect switches in metabolism between MgATP-based and MgPPi-dependent; and (v) Mg2+ affects concentrations of Ca2+ and other cations, and alleviates the effects of stress by excess [Na+]. Below, we will briefly describe each of these aspects.There are several aspects to the involvement of Mg in cell energetics: (i) Cellular ATP (and to some extent ADP) is strongly chelated by Mg2+] have significance in establishing the approximate distribution of ATP and ADP among the Mg-free and Mg-complexed forms. For instance, certain kinases, e.g., pyruvate kinase or phosphoglycerate kinase, that use ADP to produce ATP in the process of substrate phosphorylation, react with MgADP rather than ADP as their substrate [2+], these reactions will be limited because of shortages of MgADP and an excess of free ADP, which likely acts as inhibitor. On the other hand, the MgADP complex will be sensitive to changes in concentrations of Mg2+ and total ADP, both of which change reciprocally with changes in total ATP. In some instances, MgADP can act as an inhibitor, as is the case for several MgATP-utilizing enzymes in the cytosol which are competitively inhibited by MgADP [2+] is maintained at low levels, which implies that MgADP will also be low. Under hypoxia or anoxia conditions, however, cytosolic [Mg2+] markedly increases, which leads to increases in [MgADP], which may affect hexokinase activity, and thus glycolysis [2+] may impact protein kinase activities and subsequent signal transduction. Mg2+ plays an important role in the ATP binding in the active site of the kinase and facilitates phosphoryl transfer reactions  increases, as in anoxia/hypoxia. In such cases, PPi is frequently used as a substrate, as MgPPi, rather than free PPi , one group of aminoacyl-tRNA synthetases uses MgPPi and the other prefers Mg2PPi , and even small changes in intracellular [Mg2+] (below ca. 0.7 mM) may have significant effect on the MgPPi availability  ; therefolability B. Excessy UGPase . It is uproducts ,120. All2+ activates SuSy toward UDP-Glc production and it inhibits the reverse reaction (sucrose formation) , which acts as a substrate (complexed with NTP) for the pyrophosphorylases and as an effector for both pyrophosphorylases and SuSy  on both sides of a given membrane must have a crucial effect on the rates of the translocation.Transporters for Mgembranes  are moste below) . Additioe below) , the [Mg2+] between cytosol and mitochondria are the key factor in the regulation of cell respiration  in the cytosol, while ATP is mostly complexed by Mg2+ in both compartments. Depletion of Mg2+ (after growth on Mg-free media) increases free ADP concentration in the cytosol and matrix, leading to a decrease in coupled respiration and a suppression of cell growth. The [Mg2+] established under the control of AK mediates the ADP/ATP exchange between the cytosol and matrix, MgADP-dependent mitochondrial ATP synthase activity, and cytosolic free ADP homeostasis  also accompany the so-called induction phase of photosynthesis, reflecting an early response of the photosynthetic apparatus to dark-to-light transitions  increases to 1–3 mM upon the involvement of mitochondria in the reoxidation of photosynthetically formed redox equivalents via the malate valve [2+] during photosynthetic induction and steady-state photosynthesis was possible because of the data on total adenylate contents obtained by rapid fractionation of protoplasts , most AMP exists in a free form , which in turn plays a central role in the regulation of energy metabolism . Plants cofactor ,135. Theleotides , while Mree form A, and th2+] status. Under stress conditions, however, there is frequently an increase in [Mg2+] and other divalent cations, including Ca2+, reflecting lower levels of ATP produced in stressed tissues. This happens especially during anoxia (lack of oxygen), when mitochondrial oxidative phosphorylation is not working, and most energy can be acquired only via glycolysis. Under these conditions, the increased [Mg2+] leads to the activation of Mg2+-requiring enzymes and redirects the energy metabolism from ATP to PPi-utilization  due to the chelation of Ca2+ with nucleotides to nearly the same extent as with Mg2+. Magnesium allosterically activates Ca2+ binding to calmodulin, with the latter regulating target proteins in response to sub-micromolar changes in [Ca2+] [2+] in a given compartment can modulate internal [Mg2+], in a millimolar range [2+ is chelated by adenine nucleotides to nearly the same extent as Mg2+ [2+] is controlled by the AK equilibrium and that of other nucleoside kinases [2+]/[Catotal] at the same level as [Mg2+]/[Mgtotal] despite the fact that the total concentration of Ca2+ is a few orders of magnitude lower than that of magnesium [2+ when ATP level drops corresponds to an increase in internal Ca2+ [Changes in [Mgn [Ca2+] . In turnar range ,140. Ca2 as Mg2+ ; therefo kinases . This keagnesium ,143. Thenal Ca2+ .2+] that accompanies Mg2+ release subsequently leads to the activation of multiple Ca2+-regulated enzymes. These enzymes include the external NADPH and NADH dehydrogenases of mitochondria, internal NADPH dehydrogenase of mitochondria  is known to ease saline (NaCl) stress  is known to affect [Mg2+], and vice versa  can ameliorate toxic effects of aluminum ions (Al3+). For instance, the activity of the plasma membrane MGT1 transporter in rice increases upon Al addition to the roots, to prevent the Al-dependent inhibition of root elongation  in cell compartments. Mg2+, in turn, regulates Mg-dependent enzymes and controls activities of adenylate transporters, which use free adenylates , based on AK Kapp, was done for blood erythrocytes [2+] in animals. While it is unknown whether AKs in other organisms have as prominent a role in Mg2+ signaling as in plant cells, those AKs, by definition, are certainly involved in adenylate equilibrium and, perhaps, in equilibrating other nucleotides. The latter property was shown, for example, for Escherichia coli AK, which was found to have a bifunctional role as both AK and NDPK [2+] can be regarded as an unavoidable consequence of this equilibrium.Although in this review we focused on plants, we are confident that Mg eukarya —and we ahrocytes , those sand NDPK . AKs equ2+ signaling in different organisms, both eukaryotic and prokaryotic. This should also take into account organ- or organism-specific types of metabolism. For instance, the essential differences in plant and animal metabolism are partially grounded in the aspects of nucleotide equilibria, which in many animal cell types are under the buffering control of creatine kinase reaction, which is enzymatically “coupled” with that of AK., e.g., as in a muscle. In the process, creatine kinase converts ADP back to ATP, assuring fast and active energy conversion. This coupling in animal cells pushes ATP/ADP ratios to high values, affects adenylate translocation and keeps Mg2+ at low levels  upon the decrease in energy charge [2+] from the sub-millimolar to millimolar values in cell compartments, which leads to the regulation of many Mg-dependent enzymes and affects the operation of adenylate transporters. The release of Mg2+ occurs via the action of AK and triggers many processes that are regulated by [Mg2+], including the rate of photosynthesis, respiration, polysaccharide synthesis, stomatal opening, etc. The concentrations of other essential cations, such as Ca2+, Mn2+ or K+ [+], which is a feedback signal of the equilibrium of pyridine nucleotides  as a feedback signal. By means of this equilibration, AK monitors and integrates different signals to ensure energy homeostasis in response to a broad range of challenges. It acts as a powerful thermodynamic buffer enzyme [2+ signaling. The protein that at one time was considered as just a housekeeping enzyme has indeed come a long way.The unique role of AK is related to its fast equilibration of adenylates and, as it is now apparent, affecting [Mgr enzyme  that optr enzyme ,21. It a


Background: During the height of the coronavirus (COVID-19) pandemic, there was an unprecedented demand for “virtual visits,” or ambulatory visits conducted via video interface, in order to decrease the risk of transmission.Objective: To describe the implementation and evaluation of a video visit program at a large, academic primary care practice in New York, NY, the epicenter of the COVID-19 pandemic.Design and participants: We included consecutive adults (age > 18) scheduled for video visits from March 16, 2020 to April 17, 2020 for COVID-19 and non-COVID-19 related complaints.Intervention: New processes were established to prepare the practice and patients for video visits. Video visits were conducted by attendings, residents, and nurse practitioners.Main measures: Guided by the RE-AIM Framework, we evaluated the Reach, Effectiveness, Adoption, and Implementation of video visits.Key results: In the 4 weeks prior to the study period, 12 video visits were completed. During the 5-weeks study period, we completed a total of 1,030 video visits for 817 unique patients. Of the video visits completed, 42% were for COVID-19 related symptoms, and the remainder were for other acute or chronic conditions. Video visits were completed more often among younger adults, women, and those with commercial insurance, compared to those who completed in-person visits pre-COVID . Patients who completed video visits reported high satisfaction ; 13.3% reported technical challenges during video visits.Conclusions: Video visits are feasible for the delivery of primary care for patients during the COVID-19 pandemic. Shortly after its first confirmed case on March 1, 2020, New York City became the epicenter of the novel coronavirus, SARS-CoV-2, pandemic in the United States .Although video visits have been previously used and have been found to be feasible , their aNevertheless, given the need to care for patients remotely during COVID-19, the demand for video visits increased, potentially outweighing many of these prior utilization barriers. Herein we report the experiences of one large, academic, urban primary care practice with implementing a video visit program during the COVID-19 pandemic in New York City. Using a modified version of the RE-AIM Framework , we repohttps://weillcornell.org/wcima). As a high-volume tertiary-care clinic, it averages 53,000 office visits per year and serves a diverse patient population . WCIMA is a large, academic, hospital-based primary care practice of Weill Cornell Medicine and NewYork-Presbyterian Hospital  or tablet whereas providers conducted video visits through Epic Haiku (iPhone) or Epic Canto (iPad). The practice purchased iPads for providers without an iPhone or iPad to use. During or after visits, providers documented encounters using traditional Epic notes via their phone, tablet, or desktop.To minimize COVID-19 transmission and the use of personal protective equipment, starting on March 16, 2020, the majority of care at WCIMA was transitioned to video visits. Providers conducted video visits for patients with COVID-19-like symptoms, as well as for patients with acute and chronic care needs. While providers were encouraged to maximize the use of video visits, they were permitted to conduct in-person visits for urgent complaints where they deemed a physical exam was needed. To guide providers on how to conduct video visits for COVID-19 and usual care, a group of physicians developed a Video Visit Handbook . This haTo prepare for the shift to video visits, starting on March 16, 2020, clinic staff received training on how to schedule patients for video visits, including how to teach patients to download and use the app. Each week during the study period, staff and providers reviewed upcoming scheduled visits for that week, and determined who should be seen in person and who could be converted to a video visit. Those eligible for a virtual visit but without access to a smart device and internet connection were offered a telephone visit. A hybrid scheduling model was used, in which providers had half-day sessions devoted to seeing their own patients virtually and others for which they were available for video visits with any patient in the practice, to maximize access. Similar to in-person visits, video visits were 20 min in duration, but could be longer at the discretion of the provider. Video visits took place Monday through Friday, with occasional Saturday visits. Providers conducted video visits from WCIMA offices or remotely. Practice administrators worked with hospital compliance to understand billing procedures, and physicians were trained to document and bill for video visits in accordance with the new rules regarding broadened telehealth payment policies during COVID-19 from the Centers for Medicare and Medicaid Services (CMS)  of the patients seen via video, the level of service of these video visits, and the most frequent diagnoses for which these video visits were billed.To assess Effectiveness, we collected data on the proportion of scheduled visits that were: completed (as above), failed , canceled, or no-shows. During the study period, but separate from our work, the Weill Cornell Physician Organization conducted a satisfaction survey via email among patients who completed video visits within the Department of Medicine. From this survey we obtained aggregate responses from patients for video visits conducted by WCIMA providers. The survey asked patients to: (1) rate their video visit experience ; (2) report if they were satisfied with the care they received, compared with in-person visits (yes/no); (3) report what percent of care they would like to have as video visits in the future, compared to in-person (fill in %); and, (4) report if technical challenges occurred during the video visit (yes/no).To assess Adoption, we collected data on the number of staff who assisted with video visit scheduling and the number and types of providers conducting video visits (attending vs. resident vs. nurse practitioners).To assess Implementation, we collected data on the frequency with which the COVID-19 template was used and the number of iPads our clinic purchased to conduct video visits.Because this study used de-identified, practice-level data, it was deemed exempt by the Weill Cornell Medicine Institutional Review Board. As such, written informed consent was not required for participation in this study.We present absolute counts and percent frequency of occurrence. We used the Kruskal-Wallis test to evaluate differences between median age categories, and the chi-square test to compare proportions. Analyses were conducted using the software package R .In the 4 weeks prior to the study period, 12 video visits were completed at WCIMA, with 6 (50%) occurring 1 day before our study period. During the 5-weeks study period, we completed a total of 1,030 video visits. The number of these visits by week is shown in p < 0.0001]). The video visit group included more women , more Non-Hispanics , more Whites , more commercially insured patients  and fewer Medicare patients  than our baseline population. Approximately one-fourth (28%) of the video visit group were insured by Medicaid, similar to our baseline population, of which 31% are insured by Medicaid (p = 0.158).Of the 1,030 completed video visits, 817 unique patients participated. Of these, 675 patients (82.6%) had 1 video visit each whereas 142 (17.4%) had >1 video visit (range: 2–9.) The demographics of patients with completed video visits are shown in During the study period, 1,475 video visits were scheduled, of which 1,030 (69.8%) were completed and 30 (2.0%) failed due to technical problems and were converted to telephone visits. A total of 19.1% of scheduled video visits no-showed and 9.1% were canceled either by the patient or provider.SD: 0.97]) and the vast majority (94.5%) of patients were satisfied with the level of care they received during their video visit compared with prior in-person visits (SD 0.26) of future encounters with their provider be video visits instead of in-person visits. A total of 13.3% reported technical challenges during the video visit.Satisfaction data was obtained for 113 (13.8%) of the 817 patients who completed 1 video visit. Patients reported high satisfaction with their video visit  conducted these 1,030 video visits and 22 staff members helped orient and schedule patients to video visits . Among tThis study describes the implementation of a video visit program at a large academic hospital-based primary care practice in New York City during the COVID-19 pandemic. Overall, 70 providers completed a total of 1,030 video visits over a 5-weeks period, compared to 12 video visits completed in the preceding 4 weeks. Video visits increased greatly during weeks 1 and 2, plateaued in weeks 3 and 4, and dropped off in week 5, which may reflect the overall trend of health care utilization for COVID-19 in New York City during this time . Video vAnother key finding was that despite our quick ramp-up period, issues with the technology itself among those with scheduled visits appeared to be modest. For example, only 2% of initiated video visits were not completed and converted to telephone encounters. Additionally, only 13% of patients reported challenges with technology during their video visit encounter. Although we could not assess the number of patients who did not engage with video visits, our findings suggest that video visits may be more feasible than previously thought. That is, prior studies have found patients were uncomfortable with the technology and had technical issues during video visits , 19. We To our knowledge, this is one of the first studies to describe the implementation of video visits during COVID-19 in the primary care setting. There have been studies describing the implementation of video visits pre-COVID-19 across a range of clinical specialties  as well How to best deploy this equipment to patients? Which patients benefit from monitoring? How can these data be captured electronically? Will the cost of such devices will be reimbursable (Our findings not only have implications for clinical care and healthcare delivery during COVID-19, but also raise questions about the utilization of video visits in primary care moving forward. First, although we lack data on our entire sample, satisfaction scores for video visits were high and patients preferred to have half of their future visits occur via video, compared to in-person. Future research will need to determine if this preference persists after social distancing policies are relaxed. Additionally, studies are needed to assess providers' perceptions regarding the clinical effectiveness of video visits for COVID and non-COVID symptoms. Second, although video visits minimized transmission risk, they also limited the ability of providers to perform a complete physical examination and measure vital signs. Some practices, including ours, have incorporated aspects of remote monitoring into video visit encounters . Moving bursable , 26. Thibursable , 11, 27.bursable . Understbursable , 30. FinPatient satisfaction data was only available from some patients who utilized video visits, which may introduce response bias. Additionally, we do not yet have outcome data on patients with completed video visits. Finally, this study did not include the perspectives of providers and staff on implementing video visits.At the height of the COVID-19 pandemic in New York City, we implemented a video visit program at our primary care practice to evaluate and treat patients for their symptoms while maintaining social distance. During a 5-weeks period, 70 providers completed 1,030 video visits, compared to only 12 video visits completed in the preceding 4 weeks. New workflows for staff, providers, and patients were developed to implement this program. Overall, patients reported high satisfaction with the care they received during their video visits. Our findings suggest that video visits provide a feasible way to care for patients with and without COVID-19 symptoms. Additional study on the sustained implementation of video visits in primary care, as well as their effect on patient outcomes, is warranted.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.Ethical review and approval of our study was granted through the Weill Cornell Institutional Review Board. Because our study used de-identified, practice-level data, it was deemed exempt from full board review based on federal regulations. As such, written informed consent was not required for participation in this study.SS, LK, LG, and MS were involved in the study design and methods. ER, MS, and SS were involved in the data analysis. All authors contributed sufficiently to the writing and editing of this article.LK is a consultant to Mathematica, Inc. SS is a consultant for EverydayHealth, Inc. and Sharecare, Inc. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


OpenPose enable the fitting of body-pose models to the consecutive frames of a video clip and thereby make it possible to recover the movements performed by the actor in a particular video clip without the use of a point-based or markerless motion-tracking system during recording. The OpenPoseR package provides a straightforward and reproducible way of working with these body-pose model data extracted from video clips using OpenPose, allowing researchers in the fields of sign language and gesture studies to quantify the amount of motion (velocity and acceleration) pertaining only to the movements performed by the actor in a video clip. These quantitative measures can be used for controlling differences in the movements of an actor in stimulus video clips or, for example, between different conditions of an experiment. In addition, the package also provides a set of functions for generating plots for data visualization, as well as an easy-to-use way of automatically extracting metadata  from large sets of video files.Researchers in the fields of sign language and gesture studies frequently present their participants with video stimuli showing actors performing linguistic signs or co-speech gestures. Up to now, such video stimuli have been mostly controlled only for some of the technical aspects of the video material , leaving open the possibility that systematic differences in video stimulus materials may be concealed in the actual motion properties of the actor’s movements. Computer vision methods such as  Researchers in linguistics, psychology, and neuroscience who are studying sign language and gesture frequently present their participants with pre-recorded video stimuli showing actors performing manual gestures. Such gestures may be lexicalized signs of a natural sign language which can be combined to build up complex meanings  and are The reliance on video clips as stimulus materials presents researchers in sign language and gesture studies with a number of challenges. While controlling for primarily technical aspects of the video material  is rather straightforward, it is strikingly more difficult to capture aspects of the video material which are related to the actor and the specific movements they performed. For example, while the length of video clips in an experiment may be perfectly matched across the different conditions, systematic differences could nevertheless exist with regard to the speed, duration, and extent of the movements performed by the actor in each condition. In a hypothetical neuroimaging experiment on sign language, such systematic differences could for example, lead to unexpected response of parts of cortex that are sensitive to biological motion across conditions, thus distorting the actual contrasts of interest. By quantifying the bodily movements of the actor in a video clip it becomes possible to control for potential differences in these movement patterns across different conditions, or use information about velocity or acceleration as further regressors in a statistical model.Up to now, researchers have usually focused on controlling their video stimuli only with respect to certain technical properties. Some have used device-based optic marker motion-tracking systems, but only to create stimulus materials which displayed gestures in the form of point-light displays instead of a human actor . A numbeOpenPose  of the actor’s movements performed in the video. A number of existing tools may be used to quantify motion using pixel-based frame-differencing methods e.g., , but theOpenPose , 2019 deOpenPoseR, a package of specialized functions for the R statistics language and environment , whereas your choice of model of course depends on what exactly you want to control for. By default, we recommend fitting the “body25” model in the context of stimulus control. In the below example, we describe the analysis of data from the “body25” model, however, “face” and “hand” models can be fit and analyzed in the same manner. RStudio . Upon inOpenPose , 2019, we models  which caOpenPoseR on your machine is to use the install_github function provided by the devtools package:The most straight-forward way of installing install.packages(”devtools”)devtools::install_github(”trettenbrein/OpenPoseR”)R’s ability to manually install packages from a downloaded archive of the source code, however, we strongly recommend to directly install the package from GitHub.This will download and install the latest version of OpenPoseR on your system. Alternatively, it is also possible to use OpenPoseR to analyze OpenPose motion-tracking data derived from an example video clip. Throughout, we will use the results of the analysis of a short video clip showing the German Sign Language sign for PSYCHOLOGY  which make it easier to work with data derived from large sets of video files  for every frame of the input video. Consequently, an input video file with a duration of exactly 5 s and a rate of 25 frames per second will already generate 125 individual files. To make these data easier to handle, OpenPoseR combines the output for an entire video file into a single CSV file with a human-readable tabular data structure. Depending on the model  that has been fit using OpenPose the CSV file generated by OpenPoseR will contain consecutive columns for the x and y position of the points in the body-pose model followed by the confidence value (ranging from 0 to 1) for every point in the model. Every consecutive row contains the data of a frame of the input video clip. This conversion into the OpenPoseR format can be achieved using the create_csv function:create_csvThe function will create a CSV file using the JSON files stored in the specified input directory with the suffix of the body-pose model  in the given output directory. In addition, the raw motion-tracking data may be viewed by plotting a heatmap of the data using the plot_frontal function.OpenPose may contain zero values for x, y, and c of a point, indicating that the model fit failed for this particular point for the given frame. This is rather unlikely when working with stimulus video recordings which have been produced in the well-lit setting of professional video-recording facilities, but may occur with materials recorded in less optimal conditions. Similarly, points with very low confidence values  might reasonably be excluded from further analysis as these values indicate that OpenPose had trouble detecting a point in a particular frame. Including zero values and frames with very low confidence values could lead to a situation where points may appear to be “jumping around” from frame to frame due to incorrect x and y values, thereby misrepresenting the actual velocity or acceleration of the model in the video clip.On rare occasions, the model fit using OpenPoseR provides the clean_data function which will take care of this step by automatically imposing zero values as well as values below a pre-defined cutoff using the mean of the previous and consecutive frames. By using file_clean we can run this function for a large set of files:Accordingly, to increase the accuracy of our calculations the data of the body-pose model should be cleaned and thresholded before further processing. file_cleanThis will create a file called “filename_model_cleaned.csv” because the argument “overwrite” was set to FALSE upon calling the function, thereby preventing OpenPoseR from overwriting the input CSV file with the cleaned data. Using the “cutoff” argument, the threshold for the confidence value that a point has to surpass can be adjusted. If everything went well, the function will return TRUE.OpenPoseR’s file_velocity function. As already mentioned above, all functions prefixed with “file_” provide wrappers for OpenPoseR functions which make it possible to directly pass a filename to the function Notice that if your video was recorded with more or less than 25 frames per second, you will have to specify this when calling the function by setting the argument “fps” to the desired value. Output files will be created automatically and carry either the suffix “_velocity_x” or “_velocity_y.” If everything went well, the function will return TRUE.OpenPoseR, this is implemented in the en_velocity function which takes the velocity vectors of the x- and y-axis as its input. The Euclidean norm of sums for velocity as the given motion feature then is computed using the following formula:Given that our goal is to capture the total amount of bodily motion between frames in a single value, our final step in this analysis will be to compute the Euclidean norm of sums of velocity vectors . In OpenAgain, we may disregard the details of the implementation by using the wrapper function file_en_velocity which takes the filenames or path to the two CSV files created in the previous step as its input:file_en_velocityThe output file will be created automatically and carries the suffix “_en_velocity.” If everything went well, the function will return TRUE.OpenPoseR uses the functionality of ggplot2 ”plot_timeseries(example)This will create the simple plot in OpenPose for controlling their video stimuli. By quantifying the bodily movements of the actor in a video clip in terms of velocity or acceleration of a body-pose model it becomes possible to control for potential differences in these movement patterns, for example, across the different conditions of an experiment. In addition, this approach has already been successfully used to automatically detect the onset and offset for a large set of signs from German Sign Language  from large sets of video files. With its integration into the larger R environment, OpenPoseR supports reproducible workflows and enables seamless interaction with other packages in order to subject the motion-tracking results to further statistical analysis.The method and workflow presented here provides researchers in sign language and gesture studies with a straightforward and reproducible means for using body-pose estimation data derived from Language . OpenPosOpenPoseR was developed to assist sign language and gesture researchers with stimulus control in experiments that present participants with video recordings of an actor signing or gesturing, by reconstructing motion parameters of the actor using the data of body-pose models fit with OpenPose. However, we believe that the package’s functionality may also be useful for other domains of sign language and gesture research, especially as it can be continuously expanded due to the open-source nature of the project. For example, given that unlike other methods (e.g., OpenPose does not require a static camera position, is sensitive to the actual body pose of the actor in the video, and it not biased by other motion or changes in the background, the method and workflow described here has the potential to enable the use of naturalistic stimuli in sign language research, similar to the increasing popularity of naturalistic stimuli in research on spoken language (OpenPoseR may prove useful in the automated analysis of large-scale OpenPose data derived from sign language corpora (e.g., ds e.g.,  OpenPoselanguage . Similarra e.g., .https://github.com/trettenbrein/OpenPoseR.Publicly available datasets were analyzed in this study. This data can be found here: R code and curated online resources. EZ supervised the project. Both authors conceptualized the functionality of the package, implemented algorithms, contributed to the article, and approved the submitted version.PT wrote The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.