This review focuses on nutritional support in malnourished children with compromised gastrointestinal function addressing the interplay between malnutrition and gastrointestinal dysfunction, and the specific role of peptide-based enteral therapy in pediatric malnutrition. Malnutrition is associated with impaired gut functions such as increased intestinal permeability, malabsorption, and diarrhea, while pre-existing functional gastrointestinal disorders may also lead to malnutrition. Presence of compromised gastrointestinal function in malnourished children is critical given that alterations such as malabsorption and increased intestinal permeability directly interfere with efficacy of nutritional support and recovery from malnutrition. Appropriate nutritional intervention is the key step in the management of malnutrition, while alterations in gastrointestinal functions in malnourished children are likely even in those with mild degree malnutrition. Therefore, nutritional therapy in children with compromised gastrointestinal function is considered to involve gut-protective interventions that address the overlapping and interacting effects of diarrhea, enteropathy and malnutrition to improve child survival and developmental potential in the long-term. Peptide-based enteral formulas seem to have clinical applications in malnourished children with compromised gastrointestinal function, given their association with improved gastrointestinal tolerance and absorption, better nitrogen retention/ balance, reduced diarrhea and bacterial translocation, enhanced fat absorption, and maintained/restored gut integrity as compared with free amino acid or whole-protein formulas. Malnutrition remains a major public health problem and a significant cause of mortality among children under five in low- and middle-income countries (LMICs), accounting for 45% of child deaths globally –4.Given the failure of global interventions to improve nutritional status and linear growth, factors that drive and sustain malnutrition have increasingly been addressed in epidemiologic and pathogenesis research . AccordiThe gut mucosa serves as a semipermeable barrier permitting nutrient absorption and regulating immune surveillance, while retaining potentially harmful microbes and environmental antigens within the intestinal lumen by preventing their translocation across the epithelial barrier , 8. The Indeed, growth and nutritional status impairment is also possible in children with functional gastrointestinal disorders, emphasizing that gastrointestinal dysfunctional alterations may appear as a consequence or cause of the malnutrition .On one hand, malnutrition causes alterations in gastrointestinal digestive and absorptive functions . ThOn the other hand, environmental enteric dysfunction (EED) with findings suggestive of impaired gut function  is considered as an important contributor to childhood malnutrition and stunting across geographically widespread resource-limited settings , 11–13. This review will focus on nutritional support in malnourished children with compromised gastrointestinal function, addressing bidirectional interplay between malnutrition and gastrointestinal dysfunction, and the specific role of peptide-based enteral therapy in pediatric malnutrition.Gastrointestinal aspects of malnutrition refer to operation of adaptation mechanisms to protein-calorie deficiencies and the environment, as well as to the direct relationship between the degree of nutritional deficiency and the severity of gastrointestinal dysfunction .Malnutrition has been documented to be associated with pancreatic exocrine insufficiency, altered intestinal blood flow, villous atrophy, and increased intestinal permeability which eventually lead to loss of digestive enzymes, secondary lactose intolerance, loss of colonic absorptive function, and diarrhea , 10. In The alterations in digestive physiology in childhood malnutrition is considered to be a combination of two synergistic factors including (1) malnutrition-dependent impaired reabsorption of bile salts, excessive bile salt deconjugation, pancreatic exocrine insufficiency  and impaired intestinal cell function  and (2) intestinal bacterial overgrowth and frequent bouts of diarrhea resulting in impaired gut defense and thus increasing the likelihood of further alterations of the microbiota as well as the diarrhea , 16.Malnutrition is associated with thinning of the entire intestinal wall and the mucosal lining along with reduced height of the brush border and predominance of cuboidal rather than columnar mucosal cells . PermeabCorrelations between several villus morphometry parameters and measures of gut function were reported in severely malnourished children, including positive correlations between villus height (VH) and villus perimeter , between permeability marker lactulose/rhamnose ratio (L/R) and gut microbial translocation indicator lipopolysaccharide binding protein (LBP), between LBP and villus width (inflammation) as well as negative correlations between the L/R and VP and between LR and VH .Malnutrition is associated with small intestinal villous blunting that leads to reduced intestinal capacity for monosaccharide and disaccharide absorption . ConsequIn malnourished children, protein malabsorption is considered likely due to pancreatic insufficiency and small intestine villus atrophy as well as increased protein loss due to increased intestinal permeability . ProteinIn fact, any impairment in protein digestion and absorption is considered more likely in severely malnourished children and in case of concomitant diarrhea, as associated with increased total fecal nitrogen, increased protein loss from the gut or to a minor impairment in protein digestion and/or amino acid absorption . ExcessiFat absorption is markedly impaired in over 50% of children with severe malnutrition , 26. AbnNotably, the severity of steatorrhea relates directly to the degree of protein deficiency and affects triglycerides and free fatty acids, emphasizing that lipolytic activity is not the primary cause of the increased fecal fat in malnutrition . BacteriNotably, improvement in fat absorption is considered to occur concomitantly with protein repletion, reaching normality in the absence of diarrhea and after restoration of body protein , 24.Preexisting gastrointestinal dysfunction itself may also lead to an increase in morbidity and mortality secondary to malnutrition, infection, and multi-organ dysfunction if not recognized and treated accordingly .EED is used to describe group of findings suggestive of impaired gut function  in many children across geographically widespread resource-limited settings . EED, a Chronic fecal-oral transmission of pathogens is considered the principal factor underlying EED , 32. EEDAlthough malabsorption and chronic inflammation arising from microbial translocation across an impaired gut barrier is suggested as the causal pathway for the link between EED and stunting , 32, 33,The adverse impact of metabolic stress and illness on the gut integrity increase the risk of malnutrition via loss of appetite, increased intestinal permeability, diarrhea, and malabsorption of nutrients, whereas those at risk for malnutrition are already at risk for compromised gut function including mucosal intestinal abnormalities and malabsorption of nutrients and vitamins , 34.Accordingly, bidirectional interplay between malnutrition and compromised gastrointestinal function emphasizes the importance of appropriate nutritional support in malnourished children that addresses the overlapping and interacting effects of diarrhea, enteropathy, and malnutrition to improve child survival and developmental potential in the long-term , 25, 35.Most commercially available enteral formulas consist of whole protein, carbohydrates, and fat . SpecialFormulas consisting of protein in the form of free amino acids (FAA) are also available for patients with malabsorption, food allergy, or specific organ diseases . These Due to presence of impaired gut function with severe mucosal abnormalities and malabsorption in malnourished children, standard enteral formulas may not be well-tolerated . DilutioIn a past study with 400 children (aged 12–59 months) living in rural Malawi, authors reported EED in 80.7% of children and identified 77 metabolites that were either negatively  or positively  associated with gut permeability . The autNotably, supplementation with glutamine, tryptophan, and leucine has been suggested to ameliorate the mucosal pathology by increasing villus height in adults with EED and to improve barrier function when combined with micronutrient supplementation . Hence aSome studies suggested that glutamine can reduce intestinal permeability in malnourished children and those with EED , 49, whiNotably, oral replenishment of specific amino acids  is considered likely to support intestinal epithelial cell proliferation and restoration of host growth, depending on combining treatment with targeted antibiotics and/or modulators against host inflammation , 52, 53.Experimental studies indicated the association of parenteral delivery of arginine with decreased intensity of Cryptosporidium infection and the severity of growth impairment in undernourished mice through induction of defense-promoting nitric oxide pathway and the arginase pathway that is important for intestinal epithelial cell restitution , 54. TryPeptide-based formulas contain proteins that have been hydrolyzed to produce peptides of varying lengths . PresencIndeed, non-competitive specific peptide carrier systems that are independent of free amino acid carriers have been identified in the intestinal brush border, and their importance has been confirmed in patients with impaired intestinal absorption .Providing protein in a form of a protein hydrolysate is suggested to directly affect nitrogen utilization and retention, regardless of the protein quantity or the non-protein calorie-to nitrogen ratio of a particular enteral formula . ClinicaAccordingly, the development of peptide-based enteral formulas is considered a significant milestone in the advancement of clinical nutrition, being associated with improved outcomes as compared with free amino acid or whole-protein formulas in terms of improved gastrointestinal tolerance –63, nitrThis seems important given that protein hydrolysates are also considered to improve sodium and water absorption alongside nutrient absorption depending on the type of protein substrate used for the production of hydrolysate , while protein hydrolysates with a higher concentration of small peptides are also associated with a greater stimulatory effect on nutrient as well as on sodium and water absorption . Given tThe carbohydrate component of peptide-based formulas is quite similar , mainly consisting of glucose oligosaccharides, which are easily metabolized and broken down during luminal hydrolysis and mucosal absorption .The quantity of fat provided in the peptide-based formulas ranges from 10 g/L fat up to over 50 g/L fat. Although it is still controversial whether the quantity of fat affects absorption, use of medium chain triglycerides (MCT) as a fat source in all peptide-based diets (up to 40–70%) is considered very important given the likelihood of MCT to further enhance lipid absorption, particularly in patients with underlying malabsorption syndromes , 64, 65.The smaller molecular weight of MCTs than long-chain triglycerides (LCTs) facilitates the action of pancreatic lipase and thus enables MCTs to be hydrolyzed both faster and more completely than LCTs with a reduction in stool lipid excretion , 66. NotAlthough an increase in the energy density of foods and thus provision of adequate energy in diet is often achieved by increasing the lipid content , in chilUse of peptide-based formula vs. standard whole (intact) protein formula has been associated with reduced rate of tube feeding diarrhea (44.0 vs. 0.0%) in ICU patients , while sPeptide-based formulas, as compared with FAA or whole-protein formulas, were reported to improve nitrogen balance and visceral protein synthesis, to reduce diarrhea and bacterial translocation, to maintain or restore gut integrity and to improve outcomes .Immune-enhancing formulas contain specific immune modulating nutrients , while the protein composition can vary . Data fAlthough the exact pathophysiological mechanisms remain unknown, the switch from a catabolic to an anabolic state via rapid refeeding in a starved patient is considered likely to cause of the clinical manifestations of the refeeding syndrome (RFS) , 80. WhiIn fact, in addition to favorable tolerability, digestion, and absorption characteristics, peptide-based diets have also been reported to be superior over AAF, parenteral nutrition and regular oral diets in terms of weight gain and growth, reduction of the systemic inflammatory response and lower mortality rates across a multitude of patient populations . AccordiFaecalibacterium prausnitzii and Bacteroides), despite clinical improvement, particularly in those proceeded to have a subsequent sustained remission  added to enteral formulations to extend their shelflife and texture has been associated with intestinal dysbiosis by reducing the alpha diversity and microbial stability of the intestinal microflora, promoting colitis, and metabolic syndrome , 87. MorMalnutrition is associated with gastrointestinal alterations such as increased intestinal permeability, malabsorption and diarrhea , 10. EEDAAS had primary responsibility for final content. All authors read and approved the final manuscript.AAS and SE are Abbott employees. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The task of generating a semantically accurate description of a video is quite complex. Considering the complexity, of the problem, the results obtained in recent research works are praiseworthy. However, there is plenty of scope for further investigation. This paper addresses this scope and proposes a novel solution. Most video captioning models comprise two sequential/recurrent layers—one as a video-to-context encoder and the other as a context-to-caption decoder. This paper proposes a novel architecture, namely Semantically Sensible Video Captioning (SSVC) which modifies the context generation mechanism by using two novel approaches—“stacked attention” and “spatial hard pull”. As there are no exclusive metrics for evaluating video captioning models, we emphasize both quantitative and qualitative analysis of our model. Hence, we have used the BLEU scoring metric for quantitative analysis and have proposed a human evaluation metric for qualitative analysis, namely the Semantic Sensibility (SS) scoring metric. SS Score overcomes the shortcomings of common automated scoring metrics. This paper reports that the use of the aforementioned novelties improves the performance of state-of-the-art architectures.Video captioning,  The incredible success in the image captioning domain has led the researchers to explore similar avenues like video captioning. Video captioning is the process of describing a video with a complete and coherent caption using Natural Language Processing. The core mechanism of video captioning is based on the sequence-to-sequence architecture . In vide Video captioning has many applications, for example, the interaction between humans and machines, aid for people with visual impairments, video indexing, information retrieval, fast video retrieval, etc. Unlike image captioning where only spatial information is required to generate captions, video captioning requires the use of a mechanism that combines spatial information with temporal information to store both the higher level and the lower level features to generate semantically sensible captions. Although there are good works in this field, there is still plenty of opportunity for investigation. One of the main opportunities is improving the ability of models to extract high-level features from videos to generate a more meaningful caption. This paper primarily focuses on this aspect.In our paper, we propose a novel architecture that is based on the seq2seq model proposed by Most text generation architectures use BLEU  as the sFor the past few decades, much work has been conducted on analysing videos to extract different forms of information, such as, sports-feature summary , medicale.g., hierarchical approaches  task. proaches , variatiproaches , boundarproaches  etc.In earlier seq2seq literature , the decAs shown in The primary input of the model is a video sequence. The data pre-processor converts a video clip into a usable video sequence format of 15 frames before passing it to the actual model. Each converted video sequence contains 15 frames placed separated by an equal time gap. The primary output of the model is a sequence of words. The words are stacked to generate the required caption.fc7 layer’s output. This CNN layer converts each (256 × 256 × 3) shaped frame into (1 × 4096) shaped vectors. These vectors are primary inputs of our model.A video is nothing but a sequence of frames. Each frame is a 2D image with n channels. In sequential architectures, either the frames are directly passed into ConvLSTM  layer(s)uniquewordcount). So, a caption with m words is represented with a matrix of shape (m × 1 × uniquewordcount). Instead of using these one-hot vectors directly, our model embeds each word into vectors of shape (1 × embeddingdimension) with a pre-trained embedding layer. The embedded vectors are semantically different and linearly distant in vector space from others on the basis of relationship of the corresponding words.Each video sequence has multiple corresponding captions and each caption has a variable number of words. In our model, to create captions of equal length all the captions are padded with “pad” markers. The “pad” markers help create a uniformity in the data structure. The inclusion of “pad” markers do not create any change in the output as they are omitted during the conversion of tokenized words to complete sentences. A start marker, and an end marker, marks the start and end of each caption. The entire text data is tokenized, and each word is represented by a one-hot vector of shape  or frames to put more emphasis on, the stacked attention emphasizes on objects. This paper uses a stacked LSTM. Like other relevant literature  layer of this spatial-hard-pull layers determines how much effect will the sparse layer have on the context. No automatic scoring metric has been designed yet for the sole purpose of video captioning. The existing metrics that have been built for other purposes, like neural machine translation, image captioning, etc., are used for evaluating video captioning models. For quantitative analysis, we use the BLEU scoring metric . AlthougBLEU is a precision-based metric. It is mainly designed to evaluate text at a corpus level. BLEU metric can be calculated in reference to 1 sentence or in reference to a corpus of sentences . Though To get a better understanding of captioning capability of our model, we perform qualitative analysis that is based on human evaluation similar to It is to be noted that for the latter two factors, we take into consideration both the recall and precision values according to their general definition. In case of recall, we evaluate these 3 factors from our predicted captions and match them with the corresponding video samples. Similarly, for precision, we judge these factors from the video samples and match them with the corresponding predicted captions. Following such comparisons, each variable is assigned to a boolean value of 1 or 0 based on human judgment. The significance of the variables and how to assign their values are elaborated below:SgrammarSgrammar evaluates the correctness of grammar of the generated caption without considering the video.Selement•R = number of prominent objects in video•P = number of prominent objects in captionSaction evaluates the action-similarity between the predicted caption and its corresponding video, Selement evaluates the object-similarity. For each object in the caption, the corresponding Selementprecision receives a boolean score and for the major objects in the video, the corresponding Selementrecall receives a boolean score. The average recall and average precision is combined to get the Selement.As SactionSaction evaluates the ability to describe the action-similarity between the predicted caption and its corresponding video. Sactionrecall and Sactionprecision separately receives a boolean score  for action recall and action precision respectively. By action recall, we determine if the generated caption has successfully captured the most prominent action of the video segment. Similarly, by action precision, we determine if the action mentioned in the generated caption is present in the video or not.Combining equations During this research work, the SS Score was calculated by 4 final-year undergraduate students studying at the Department of Computer Science and Engineering at the Islamic University of Technology. They are all Machine Learning researchers and are fluent English speakers. Each caption was separately scored by at least two annotators to make the scoring consistent and accurate.Tensorflow 2.0, Platform: Google Cloud Platform with a virtual machine having an 8-core processor and 30GB RAM, GPU: None. We used the Microsoft-Research Video Description (MSVD) dataset  attention: In relevant literature, though the use of attention is very common, the use of stacked attention is quite infrequent. •Triple Attention: Since the use of dual attention has improved performance in comparison to single attention, it is only evident to create a triple attention to check the performance. Considering our limitations, our stacked attention gives satisfactory results for both BLEU and SS Score in comparison to the commonly used attention methods when performed on similar experimental setup. Graphs in To boost the captioning capability, some state-of-the-art works like By performing various trials on a fixed experimental setting, we analysed the spatio-temporal behaviour of a video captioning model. After seeing that single layer encoder LSTM causes more repetitive predictions, we used double and triple layer LSTM encoder to encode the visual information into a better sequence. Hence, we were able to propose our novel stacked attention mechanism with double encoder layer that performs the best among all the variations of LSTM layers that we tried. The intuition behind this mechanism is that, as our model separately gives attention to each encoder layer, this generates a better overall temporal context for it to decode the video sequence and decide whether to give more priority to the object or the action. And the addition of Spatial Hard Pull to this model bolsters its ability to identify and map high level semantic visual information. Moreover, the results also indicate that addition of excess SHP units drastically affect the performance of the model. Hence, a balance is to be maintained while increasing the SHP units so that the model does not over-fit. As a result, both of these key components of our novel model greatly contributed to improving the overall final performance of our novel architecture, that is based upon the existing fundamental concepts of state of art models.Although the model performed good in qualitative and quantitative analysis, our proposed SS Scoring method provides greater insight to analyse video captioning models. The auto metrics, although useful, cannot interpret the videos correctly. In our experimental results, we can see a steep rise in the BLEU Score in Video captioning is a complex task. This paper shows how stacking the attention layer for a multi-layer encoder makes a more semantically accurate context. Complementing it, the Sparse Sequential Join, introduced in this paper, is able to capture the higher level features with greater efficiency.Due to our computational limitations, our experiments use custom pre-processing and constrained training environment. We also use a single feature as input unlike most of the state-of-the-art models. Therefore, the scores we obtained in our experiments are not comparable to global benchmarks. In future, we hope to perform similar experiments with industry standard pre-processing with multiple features as input.The paper also introduces the novel SS Score. This deterministic scoring metric has shown great promise in calculating the semantic sensibility of a generated video-caption. However, since it is a human evaluation metric, it relies heavily on human understanding. Thus, a lot of manual work is to be put behind it. For the grammar score, we can use 10.7717/peerj-cs.664/supp-1Supplemental Information 1Each sheet corresponds to a different model architecture. The name of each sheet denotes the architecture details. In each sheet, each row corresponds to a particular sample from the validation set, and each column represents the corresponding epoch. The values in each cell are the corresponding SS scores.Click here for additional data file.


In recent years, interest in video game live streaming services has increased as a new communication instrument, social network, source of leisure, and entertainment platform for millions of users. The rise in this type of service has been accompanied by an increase in research on these platforms. As an emerging domain of research focused on this novel phenomenon takes shape, it is necessary to delve into its nature and antecedents. The main objective of this research is to provide a comprehensive reference that allows future analyses to be addressed with greater rigor and theoretical depth. In this work, we developed a meta-review of the literature supported by a bibliometric performance and network analysis (BPNA). We used the PRISMA  protocol to obtain a representative sample of 111 published documents since 2012 and indexed in the Web of Science. Additionally, we exposed the main research topics developed to date, which allowed us to detect future research challenges and trends. The findings revealed four specializations or subdomains: studies focused on the transmitter or streamer; the receiver or the audience; the channel or platform; and the transmission process. These four specializations add to the accumulated knowledge through the development of six core themes that emerge: motivations, behaviors, monetization of activities, quality of experience, use of social networks and media, and gender issues. Video games go beyond being a simple hobby linked to young people and adolescents; instead, they have become a new form of adult leisure activity commonplace in contemporary society . They cosocial live streaming services (SLSSs) to refer to a new type of social network with specific features such as synchronization, real-time streaming, viewer–streamer interaction, and a rewards system. For their part, Ma et al. [a form of media integrating the public, communities, interaction, and passivity, while bridging the gap between online games and traditional video media (such as TV)”, and regarding the content, stated that “video game live streaming is defined as a network broadcast with ‘online games’ as the specific content” [The first academic papers on video game live streaming were relatively new. Li et al.  and Hamia et al.  underlina et al.  referredcontent” . Besides the definition or terminology used, there are certain common features that have emerged in most of the related publications to date. One is the importance of the process of communication for understanding the phenomenon of video game live streaming; another is the central role played by the activity of video games, whether active participation—development and streaming—or passive—watching without taking part . The preThree complementary methodological approaches were combined in order to achieve the proposed research objectives. First, we applied the PRISMA protocol ,24 desigThe PRISMA protocol  is a proIdentification: The present study used a single source of information, Web of Science (WoS). WoS is a data source owned by Clarivate Analytics that includes several databases; in our work, we used the Core Collection and the citation indexes: Science Citation Index Expanded (SCI-EXPANDED) 1900–present; Social Sciences Citation Index (SSCI) 1956–present; Arts & Humanities Citation Index (A&HCI) 1975–present; Conference Proceedings Citation Index-Science (CPCI-S) 1990–present; Conference Proceedings Citation Index-Social Science & Humanities (CPCI-SSH) 1990–present; Book Citation Index-Science (BKCI-S) 2005–present; Book Citation Index-Social Sciences & Humanities (BKCI-SSH) 2005–present; Emerging Sources Citation Index (ESCI) 2005–present. Although this represents a limitation in terms of the coverage of documents in the review, it ensures a single format for citation, usage, and classification in areas of research. WoS offers an adequate volume of documents, proven scientific quality, and data cleaning to correct errors . The selScreening: The resulting searches were refined and filtered using the term “video game” and its plural. The titles, keywords, and abstracts of the documents were reviewed. At this initial stage, documents that indicated that the research was about live streaming were kept and those that clearly did not address the specific topic, gaming or video games, were removed. Records that could not be discarded without a full-text reading were retained. The references contained in the main studies were tracked and the results cross-checked against previous reviews ,23. ThisEligibility: The criteria applied were, first, that the main topic of the document directly addressed video game live streaming, even if it was as part of a broader study; second, that the document was indexed in the WoS and had undergone some type of peer review process as an article or text from a conference, or was editorial material indexed in the WoS; and third, that it was possible to retrieve the full-text document. All references resulting from the application of these criteria were downloaded. Included: The full-text documents were read excluding those linked to live streaming that did not specifically deal with games or video games, those whose main topic was not clear, those that did not present any research data, or those whose methodology could not be deduced specifically from the text. The search for and reading of references yielded 111 documents. The process started in September and was continuously updated to include new records indexed in the WoS until 1 December 2020. From the full-text reading, the different definitions provided for the phenomenon of video game live streaming were extracted ,18,19,33Documents were classified according to the main focus of the research. In line with previous studies ,23 and bThe study was based on a number of indicators: direct citation, h-index of the categories considered  as well as usage counts since 2013 and in the last 180 days. Citation is used as a measure of the relative influence of publications; when a document is highly cited, it can—with the appropriate caveats —be consielectronic sport, esports, and e-sport game were all introduced under the word e-sport. The other parameters used in VOSviewer are detailed in Science mapping produces a spatial representation of the relationships among a group of documents, authors, journals, disciplines, or research domains. The analysis of scientific networks has previously been used in different fields to visualize these types of relationships ,42. In l“thought or discourse communities, which are parts of society’s division of labor” [This study adopted a broad definition of research domain: f labor” . As Hjorf labor”  argue, rA research domain arises from one or more academic fields that expand according to interests, themes, values, or conflicting points of view or values, with the academics involved moving toward specific sub-communities ,48. In oWith regard to the essence of the phenomenon—the “what”—the application of the PRISMA protocol  yielded As for the “how”, the domain of the accumulated research on video game live streaming is constructed through the application of various theoretical frameworks. Although many of the papers did not provide information on the theories they reference and are broadly based on previous studies, it is possible to identify certain established frameworks that are repeatedly used. Within the overall domain, the four above-mentioned subdomains or areas of specialization can be distinguished. The group of documents or subdomain focused primarily on the streamer employed well-established theories such as self-determination theory ,54, affoRegarding the discipline and the methodology, The “who” and the “where” were answered by examining the main players through the bibliometric analysis of the 111 documents in the review. Although the period of analysis started in 2010, the first studies did not appear until 2012. These were technical papers focused on developing new systems that allow video to be shared with players online , lab expIn From 2017, the number of publications grew exponentially before levelling off at an average of around 20 documents per year. The research generated has attracted attention from other areas of knowledge such as psychology ,60,65,71To put these values into context, it should be noted that although the audience subdomain registered the highest values, it was also the specialization that accounted for the greatest number of documents belonging to disciplines aligned with the study of behavior ,59,61. TMoreover, the sample of publications was composed of a large number of proceedings papers, a clear indication of its relative recency. We should expect to see a significant proportion of these preliminary articles evolving and being refined ,139,140.The WoS categories also pointed to a gradual increase in the number of research areas involved. It should be borne in mind that the same document can be classified in more than one category depending on the indexing of the journal or the conference in which it appears. Interestingly, despite the disruption to the video game industry  caused bAfter using the VOSviewer thesaurus file to clean the keywords, we were left with a total of 390, of which only 85 met the established criterion of a minimum of two occurrences. The software calculated the total strength of the co-occurrence links between terms, and selected those with the highest total link strength. The full list, along with the thematic clusters, links, and occurrences, is presented in The thematic clusters related to motivation (C1), behavior (C2), monetization of activities (C3), and media usage (C5) predominated in the areas of specialization focused on the streamer and the audience. Furthermore, they were the ones that had had the greatest impact in terms of citations and downloads. Conversely, the thematic clusters focused on the quality of the experience (C4), and gender and content (C6) had a stronger presence in the subdomain focused on transmission. While their impact has been less marked, it is expected to increase as connections develop between the accumulated knowledge in the different specializations. Finally, documents covering the channel or the transmission platform appeared across the six thematic clusters identified, and achieved an average impact relative to the other specializations detected. Subsequently, the main topics on video game live streaming obtained in each cluster are detailed. The first cluster contains topics relating to the motivations for watching and streaming, and to the consumption of e-sports, work, and the digital economy. This cluster includes studies focused on the different motivations that drive the audience ,17,64 anThe second cluster lies close to the first and mainly deals with the behavior of users and streamers. The link with the first cluster is established through the different motivations that give rise to widely differing behavior . The topIn the third cluster, which is aligned with the first two, the topics of the domain extend to the monetization of the activities of streamers and platforms ,118. It The fourth cluster brings together topics related to the quality of experience (QoE) ,94,141 aThe fifth cluster is linked to the social component of video game live streaming, media usage, and social media. The relationship with the first three clusters  relates to the terminology used by Gros, Wanner, Hackenholt, Zawadzki, and Knautz  and ZimmThe sixth cluster includes the most cross-cutting and current issues detected in all clusters, most notably the technology, the culture generated around live streaming communities, and more controversial matters related to gender, sexism, sexual content, and regulatory policies. In relation to the topic illustrated by the keyword “technology”, it can be seen that it is overly general; however, this node appears aligned with some of the documents on the professionalization of streaming content ,54. The This article presents an exhaustive literature review carried out using the PRISMA protocol. The combination with BPNA and the science mapping of the most significant research topics represents the first attempt at a comprehensive approach to video game live streaming research. After reviewing 111 documents, the results pointed to the communication process as the central axis around which the study of this new mass phenomenon revolves. It was thus possible to adopt the elements of the model proposed by Shannon  to identRelating to the audience, three basic dimensions have been studied: information, entertainment, and socializing ; these rRegarding the streamer, we found studies exploring the characteristics, behaviors, and challenges of live programming, revealing that streamers are motivated by the desire to share knowledge, socialize, and build identities online, taking on challenges related to the tools available and maintaining viewer engagement . In thisRelating to the research on platforms, it was shown that live streaming has become a novel hybrid of conventional television, YouTube, and a social network, whose main (though not entire) content is video games. As Chen and Xiong  revealedFinally, as far as streaming itself is concerned, the possibilities are extraordinary. In the near future, a great deal of attention will be paid to the subscription models for cloud gaming platforms such as Stadia, Xcloud, or Amazon Luna, which offer catalogues of video games and incorporate functionalities that further facilitate live streaming. The management of communication and the broadcast of messages in group chats  pose newThis review underlines the complex, multidisciplinary nature of the phenomenon of video game live streaming. The differences in citation patterns, methodologies, and theoretical frameworks used may stem from the different areas of research involved in each of the identified specializations, although they denote common characteristics of the research domain. Strong links were observed between subdomains centered on the streamer and the viewer; however, they appear to be somewhat disconnected from the specializations focused on the platforms and transmission processes. This disconnect will no doubt be resolved in the future as the volume of research increases and the related areas of knowledge are further democratized. The review of the documents in the sample provides a rigorous, detailed definition of live streaming services. This approach highlights the importance of gaining a greater theoretical understanding of the communication process and the interactivity entailed in this type of service. The review provides a significant reference for scholars interested in examining this new phenomenon. “Communication includes all aspects involved in the creation, export, import, and processing of artifacts used to link objects in the world. The study of communication encompasses all features of a communication system, including its technical, personal, social, and organizational forms”, very much present elements in the live transmission of video games. Although research on the live streaming of video games is awakening academic interest, the theoretical contributions developed are still limited, from its definition of the phenomenon and the elements involved, to its meaning in a new cultural and media context. The approach proposed here does not pretend to be unique or exclusive; it has attempted, through the compilation of accumulated knowledge, to show the basis for future research. Future works will need to incorporate new theoretical approaches from the field of videogame studies (e.g., the fit that the discussion between narrative or ludology has for research on live streaming , or the However, the results presented here are not free from limitations, which relate to the studies included in this review. The first limitation stems from the use of a single data source to obtain the final sample, which means relevant documents not indexed in the WoS were excluded. Although this was a self-imposed limitation to avoid mixing citation formats or subject categories, it opens the door to new analyses with other databases that can help test the findings obtained here. Second, although a rigorous procedure was employed in order to avoid possible biases, the coding of documents and their classification inevitably entails a certain degree of subjectivity that could distort some of the assessments. However, we believe that the combination of methods helps to minimize the impact. Third, this study presents a snapshot of the situation; the evolution of the research domain thus remains to be studied, calling for future reviews that can capture the dynamism of all the issues presented here.We conclude that there is still a great deal of heterogeneity in the results obtained in the different thematic clusters and specializations identified. Given the increasing academic interest in live streaming services, there is a need to create a common basis for research that can guide theory and practice and enable interdisciplinary collaboration to support the future growth of this emerging research domain.


You Tube is one of the most commonly used online sources for sharing information and knowledge. Academic topics or clinical data shared on this platform is not peer reviewed or evaluated by subject specialists for accuracy. No study was found in the literature examining the validity of crown preparation videos available at this platform.  To evaluate the authenticity of the content and quality of the crown preparation videos uploaded on the YouTube.  The systematic search for YouTube videos was carried out over a period of one year from January 2020 until February 2021. The keywords or phrases and tags used were crown preparation, PFM crown preparation, all ceramic crown preparation, and dental crown preparation. The videos were shortlisted on the basis of inclusion and exclusion criteria to select educationally useful videos in terms of content and quality.  Three subject specialists evaluated the videos on crown preparation three times to shortlist only 12 (11%) educationally useful videos out of 109 relevant videos. These 12 videos met the preset inclusion criteria.  Although YouTube is the most popular social media platform used as the source of information by the students, the majority of uploaded content lacks authenticity. This study found that crown preparation videos uploaded by the faculty members or subject specialists can be considered as the reliable source. The undergraduate dental education, as with any other educational program, needs to adapt evolving techniques and materials in the curriculum in order to fulfill the changing needs of the dental practice . Albeit The predoctoral dental syllabus generally targets the development of psychomotor dexterity of students in their initial clinical years . This apThe preference of video-assisted clinical instruction in dentistry (VACID) over conventional education for learning practical skills has been reported in some studies , 8. AkhlTowards the end of 2019, the coronavirus disease surfaced as an acute respiratory infectious disease in China which was later declared by the World Health Organization (WHO) as a pandemic with high fatality and morbidity rates . The preDuring these situations, YouTube audio-visual recordings became convenient and an accessible alternative to the study materials , 15 and Therefore, the aim of this study was to evaluate the quality and reliability of the information provided in the crown preparation videos currently available on YouTube. No study was found in the literature that has reported the efficacy of YouTube videos on the topic of crown preparation.The research was conducted as a cross-sectional analysis. The systematic search for YouTube videos was carried out over a period of one year from January 2020 until February 2021. The keywords or phrases and tags used were crown preparation, PFM crown preparation, all ceramic crown preparation, and dental crown preparation.http://www.youtube.com” using the google chrome web browser. Inclusion and exclusion criteria helped in shortlisting only 12 videos out of 109 available videos.The following data was collected about the videos: name of the uploader and publisher of the video, video title, YouTube video links, total number of the views of the video, total number of likes or dislikes of a video, total number of positive and negative comments (if any) related to the video, date of uploading, total length of the video, video quality, sound quality, the prepared tooth by using the FDI notation system, type of preparation, the type of prepared tooth, qualification of the presenter, primary or permanent tooth, the language used by the presenter, and the quality of English language. For data collection, we explored “2.3.All videos that fulfilled the predetermined criteria were included. Content must be relevant and scientifically valid reflecting an acceptable knowledge about crown preparation. The videos must include crown preparation  as well as presented (if there is a presenter) in comprehensible English language. The videos presented by the faculty members and on tooth preparation done on permanent teeth  were included. The authors agreed to include videos that followed the ideal crown preparation parameters. The contents regarding crown preparations to be present in the videos were expected to be based on standard guidelines as described in standard text books for fixed prosthodontics and recently reported in the literature –18. The British as well as American accent, both were considered acceptable for inclusion in the study. Minor variations were ignored. If the video was in the English language  and the purpose was clearly understood, it was included in the study. The main factor to be assessed was the comprehensible English language. Senior faculty members who have been involved in teaching the subject in English for five or more years were involved in the assessment of videos.Contemporary Fixed Prosthodontics and Fundamentals of Fixed Prosthodontics, were kept as the standard [Out of the available videos, poor quality videos  and non-English language videos were excluded. Among the videos with exactly the same or duplicated content, the videos with lower views were excluded. Videos with more than 5 negative comments were excluded. Other videos in the exclusion criteria were the ones providing information without scientific justification. Two of the most commonly recommended books for fixed prosthodontics, standard , 18.t-test was applied to find out the t-value and determine significant differences. Pearson's correlation coefficient (r) was used to find a correlation of like versus dislikes. P value of less than 0.05 was statistically significant at 95% confidence interval.The data gathered from YouTube videos was summed up utilizing a standard form and entered in Microsoft Excel 2016. Data analysis was performed on SPSS version 24.0. Descriptive statistics like mean and standard deviation were used to present the output of analyzed data. For the analysis of variance, statistical After initial evaluation of the 109 shortlisted YouTube videos by three subject specialists, 42 (38.5%) videos were considered as relevant and were in the English language. Out of these 42 videos, 30 (71.43%) videos were excluded due to poor sound or picture quality, not presented by a faculty member, tooth preparation was not done on permanent teeth, negative comments were more than five in number, or the information about crown preparation was not adequate. Only 12 out of 109 videos (11%) were found to be educationally useful and possessed relevant information about crown preparation, and one out of these 12 videos was animated . The lint = −3.63, P = 0.02). The average number of “likes” for videos by the viewers/day was 1.12 (SD = 1.62). Probably because login information is required to “like,” “dislike,” or comment on any YouTube video, so most of the users just view the videos and cannot mark on “like” or “dislike” and also could not make any comments.Total mean duration of these educationally useful videos was 971.67 s (SD = 459.66) . The meat = −0.97 and P = 0.35).The mean of “dislike” of educationally useful videos was 5.92 (SD = 9.25) which also showed that not much difference of opinions was present with the comment or not comment . Likewise, a significant correlation was found between the total views and the like/day , a significant correlation was found between the total views and the positive comments/like , significant correlation was found with video merit , but no correlation was found with video scores  , and a significant correlation was also found between the total likes and the like/day . However, no correlation was found between the video scores  and video merits . Also, a significant correlation was found between the total dislikes and dislike/day  as presented in The correlation between the total 12 educationally useful videos, the number of views, and the number of viewers/day was positive and high, and the result was statistically significant (This research offers the evaluation of already available YouTube videos on the topic of crown preparation with particular focus on the quality and information in the videos and viewers' engagement, upgrading the knowledge of students to promote efficient learning.The evolution of digital platforms for learning has led the instructors to think diversely and to upgrade the learning and teaching strategies . In the The authors believe that this is the first report on the usefulness of YouTube videos on the topic of crown preparation. In this study, a detailed evaluation of these videos and information presented is carried out to determine their educational value. It was observed that a good number of recordings related to the crown preparation were created and uploaded anonymously with no details of the creator whether the content creator is a faculty member or not , 19. AlsThis study also observed that the viewers who liked the video also dropped a positive comment, indicating that if the information being delivered is correct, from some reliable source, and contributes to learning of the viewer, such a video will generate more comments and likes. Similar observations were reported by Ahmad and colleagues . This stGenerally, the studies show support for video-based learning as it is considered an efficient learning tool –23. MoreOur research showed that the videos on YouTube that were considered useful based on likes and dislikes totaled the video score of 76% . This meOur study confirms the findings of other studies , 22 thatThe merit score of crown preparation videos as devised from positive or negative comments was 38.9%. The negative remarks on the videos were mostly addressed to further improve the quality and to use accurate and relevant keywords for search of the video , 25, wheThe authors believe that the topic of “crown preparation” is appealing only to the dental community. Hence, the YouTube videos identified and included in this study were not very high in number. Another limitation is that the content assessable on YouTube varies, and recordings are uploaded and removed constantly. Subsequently, the results may change depending upon the time period of search. In the future, the research can be done to adopt a longitudinal or field-based approach to evaluate the value of YouTube as a mode of learning for students. For further improvement in video-based distant learning, virtual simulators can be used along with video-based education to replicate the clinical condition and enhance practical skills. This will not only provide learners with a realistic situation but also give feedback to both, the instructor and students .When used for educational purposes, YouTube can enhance the learning experience. It provides enhanced auditory and visual learning by use of eye-catching animations, videos, and pictorial explanation of content. Although the practical skills can only be learnt on simulators, extracted teeth, or on patient, but background knowledge plays a key role in accurate performance of the procedure. The viewers should be vigilant about the origin of the information being watched to avoid any misleading information. Moreover, the content creators should be more responsible when creating the videos to pass on the authentic knowledge from the educational point of view to make online learning via the YouTube platform more beneficial.


To provide a preliminary test of the prediction that fewer former heavy drinkers will identify themselves as being in recovery in the UK versus the USA.An online cross-sectional survey was completed by a convenience sample of former heavy drinkers. This sample was identified from participants recruited to complete a questionnaire about alcohol consumption. The recruitment advertisement specified that the participants did not need to drink alcohol. The survey included items assessing self-reported current and past levels of alcohol consumption, alcohol dependence at time of heaviest alcohol consumption (ICD-10 criteria), and questions regarding identifying as currently or ever being in recovery taken from a survey by Kelly et al. (2018).p = .841 and 0.300 respectively). Compared to participants from the UK, participants in the USA were more likely to report that they had a problem with drinking but now no longer do , and that they currently identified  or ever identified  as being in recovery.Out of 5002 participants who completed the questionnaire, 150 were identified as former heavy drinkers from the UK or the USA. The proportion of participants reporting alcohol dependence, and the proportion of participants reporting past year abstinence, did not differ significantly between the UK and the USA (Identifying as being in recovery appears more common in the USA than the UK among former heavy drinkers. This apparent difference in prevalence may reflect historic differences in treatment services offered in these countries, particularly with respect to the predominance of a 12-step approach in the USA. These findings should be replicated in a representative sample. The term, ‘in recovery’ is used by the Alcoholics Anonymous (AA) movement to describe the ongoing need for vigilance to maintain sobriety  and to prevent a relapse to hazardous drinking . The perWhile AA and the 12-step movement have a global reach, they largely originate in the USA  . As the The current pilot study presents preliminary analyses to test the prediction that the recovery identity is more common amongst former heavy drinkers in the USA as compared to the United Kingdom. The study is regarded as a pilot primarily because the data is not representative of the general population. Nevertheless, the sample is adequate to provide a preliminary test of this prediction and to establish whether more research on this topic might be merited.The survey was conducted through the Prolific website and employed an advertisement asking for participants to take a short survey (approximately 15 min) about drinking alcohol . The advThe survey also included questions regarding current and past alcohol consumption that allowed us to identify participants who were former heavy drinkers using a set of criteria employed in previous research. Briefly, former heavy drinking was defined as endorsing having ever drunk five or more  drinks at least once a week for a month or longer, when participants were drinking at their heaviest. Of these participants, only those who reported no alcohol consumption in the past year (abstinent), or reported alcohol consumption of a lower risk , were included in the former heavy drinking sample and only this subgroup was examined in subsequent analyses. The definition of current moderate drinking was: (1) usual consumption of two or less drinks per drinking day; (2) consuming five (six) or more drinks on one occasion less than once per month; and (3) having never consumed more than seven drinks (eight in the UK) on one occasion in the past 6 months –14.Participants identified as former heavy drinkers were then asked several items taken from the Kelly et al.  survey sAt the end of the survey, and after answering questions asking about demographic characteristics, participants were asked whether they answered all the questions truthfully . In addition, an attention check question was nested in a series of questions early in the survey, “I want to indicate that I have read this question by checking all the time?”  . Only thp > .05) in demographic characteristics between participants from the UK or the USA . There were no significant differences (p = .300). Further, there was no significant difference in the proportion of participants who met criteria for lifetime alcohol dependence (p = .841). However, compared to those in the UK, participants residing in the USA were more likely to state that they used to have a drinking problem on both the yes/no former problem question  and on the rating of severity of their drinking at its heaviest as a major or very major problem . Finally, compared to UK participants, those in the USA were more likely to identify as currently  or ever  being in recovery.Table Participants from the USA were more likely than those from the UK to endorse that they used to have a problem with their drinking but now no longer do, and to regard themselves as currently or ever being in recovery. One possible explanation for this difference is that participants from the US may have had more severe alcohol use symptoms  than those from the UK. This explanation cannot be ruled out using the current data. However, the proportion of participants who met criteria for alcohol dependence was similar in both the UK and the USA, reducing the likelihood that differences in the severity of alcohol consumption was the sole reason for the observed findings.An alternate explanation for these findings is that these differences are not due to, or at least not entirely due to, differences between the two countries in the severity of alcohol consumption of participants prior to reducing to abstinence or moderate drinking. Mainly, recovery-oriented treatment services have been more common in the US than in the UK –9. GivenThere are a number of limitations associated with the current data set that are important to address in order to strengthen the conclusions suggested by these analyses. Primarily, there is a need for a larger sample of former heavy drinkers in order to allow for the exploration of the association of demographic and drinking severity characteristics with identifying as being in recovery. Further, this larger sample should ideally be collected using methods that allow for confidence that the results are representative of the general population of former heavy drinkers in each country. In addition, there would be considerable worth in incorporating a qualitative component to such a study in order to establish what participants mean by being in recovery (or not) and whether this meaning varies by culture . This quA detailed understanding of the factors associated with a willingness to adopt a recovery identity could have considerable worth in informing ongoing policy decisions regarding the best ways to attract people with hazardous alcohol consumption to seek help and to then provide them with engaging tools to promote the successful resolution of their drinking concerns. While the current data provide preliminary support to suggest that former heavy drinkers in the US and UK may significantly differ in their identity as being in recovery, further work is necessary to generalize results at a population level. Additionally, future work should examine how different characteristics of previous heavy drinking  impact the recovery identities of former heavy drinkers in the USA and the UK.


Affective computing has suffered by the precise annotation because the emotions are highly subjective and vague. The music video emotion is complex due to the diverse textual, acoustic, and visual information which can take the form of lyrics, singer voice, sounds from the different instruments, and visual representations. This can be one reason why there has been a limited study in this domain and no standard dataset has been produced before now. In this study, we proposed an unsupervised method for music video emotion analysis using music video contents on the Internet. We also produced a labelled dataset and compared the supervised and unsupervised methods for emotion classification. The music and video information are processed through a multimodal architecture with audio–video information exchange and boosting method. The general 2D and 3D convolution networks compared with the slow–fast network with filter and channel separable convolution in multimodal architecture. Several supervised and unsupervised networks were trained in an end-to-end manner and results were evaluated using various evaluation metrics. The proposed method used a large dataset for unsupervised emotion classification and interpreted the results quantitatively and qualitatively in the music video that had never been applied in the past. The result shows a large increment in classification score using unsupervised features and information sharing techniques on audio and video network. Our best classifier attained 77% accuracy, an f1-score of 0.77, and an area under the curve score of 0.94 with minimum computational cost. Life without music is hard to imagine because it is deeply associated with our society, culture, and mental satisfaction. Music is a source of pleasure which has psychological benefits including stress management, and improved mood and motivation. Some studies5 have investigated the question “why do we listen to music?” and found that music boosts our mood, feeling, and self-confidence. Additionally, music is an extremely safe, non-invasive, easily accessible, and non-expensive media.Human history shows the importance of music in human lifeA music video is a commercial product containing music with imagery to promote a song or album. Modern music videos use a wide range of filmmaking styles that are used as a marketing tool to promote the sale of music recordings. Many music videos present specific images and scenes from the song's lyrics, while others take a more thematic approach. To produce a music video, the director tries to create a visual depiction that corresponds to their subjective interpretation of the emotion expressed in a piece of music.In the digital age, social media and music streaming services have increased the demand for music videos. Most music video retailers sell emotion as a piece of music via online and offline businesses. Industrially, music emotion analysis has been used for personalized and generalized music recommendation; emotion-based music searches; automatic music creation; and more. Other applications of music’s affective computing are mental health treatment, the gaming industry, advertisement, and broadcast stations.The major challenge in music’s affective computing is data annotation because emotion has an inherent relationship derived from multiple aspects of music which vary over time. Emotion description is inherently subjective because the context of music corresponds to the associated emotion. The presentation of emotions also varies across cultural groups, languages, and intentions of a music maker. Music emotion itself is a complex representation that combines the vocalist’s emotion, the melody, and linguistic information. Music videos contain further complexity because visual and acoustic information have distinct emotion representation schemes. One possible approach to address this complexity when predicting emotion in a music video is to separately analyze the audio and visual information, and then integrate the results.6 was used in the proposed architecture for run-time information sharing and boosting. Finally, unsupervised networks were fine-tuned with an upgraded version of the music video emotion dataset of7. The key contributions of this study are listed below:We propose an unsupervised multimodal architecture for music video feature representation using a large unlabeled dataset. Further, we fine-tuned the multimodal with a supervised music video emotion dataset of six emotion categories.The complexity of the neural networks using conventional 2D/3D convolution was reduced using the channel and filter separable convolution. The proposed convolution minimized the training parameters and even improved the performance. The light weighted representation allows us to train the autoencoder of music and video using an end-to-end manner.8 strategy was applied for unsupervised music video emotion representation with MMTM module to boosts and shares the learned information of audio and video at the training time of the network.The slow–fast networkIn this paper, we propose an unsupervised multimodal method for music video emotion classification. An autoencoder network with music and video information was trained on music video samples collected from the Internet without any label. Music and video multimodal network architectures were trained on more than 0.13 million music video samples for joint information representation. Initially, general 3D convolutions were used in our multimodal architecture that drastically increases the network parameters. The system complexity was reduced by introducing channel and filter separable convolution. Similarly, the complexity in the music networks using a 2D square filter  is a sub-domain of human affective computing that uses face geometry and textual input to recognize human emotion. The majority of the studies in FER35 relied on facial context information and dominant facial landmarks. The quality of the face image, the camera distance, and the depiction of facial landmarks are the most important tuning parameters for FER. Yagya et al.36 proposed a supervised technique for music video emotion analysis that incorporates audio, video, and facial data. The study investigates the importance of musical elements for affective computing in music videos, and the quality of the face image is important to determine the overall music video emotion. SmallBigNet32 presented a method for dealing with the various perspectives in video, in which a big view directs a small view branch in 3D feature space. The works in the video domain that were discussed were only for supervised classification. Research has also been conducted for unsupervised video classification39 using convolution and recurrent neural networks. Nomiya et al.40 proposed unsupervised emotion classification in video using the Gaussian mixture model (GMM).As far as we know, no research has been done on video emotion analysis in the context of music video content. We'll go through some of the supervised and unsupervised video processing techniques that have been created. Our model is based on some prior arts that were used in video analysis. Supervised video classification is widely used in computer vision41, motion boundary histograms42, and trajectories43 for action recognition in video data. However, it is methodologically unsatisfactory given that optical flow is a hand-designed representation, and two-stream methods are often not learned end-to-end jointly with the flow. The Slow–fast network in6 is based on the idea of preserving the spatial and temporal information of video using end-to-end learning. The slow branch captures static but semantically meaningful features whereas the fast branch captures the temporal information of the video sequence. Xiao et al.44 demonstrated that a slow–fast video network with audio improves the video action classification and detection task, but they only evaluated audio on the fast pathway. In this study, we extend this concept with audio and video in both slow and fast path and the learned information of both branches is boosted and shared using the MMTM module. The MMTM shares the information from two modalities and boosts the information using the squeeze and excitation45 method.Many supervised studies have used hand-crafted spatiotemporal features based on optical flow to capture the motion information of video. This method typically includes histograms of flow47 and deep learning approach5. At the time of writing, the authors are not aware of any unsupervised methods for music video emotion classification. We provide an unsupervised music video representation approach as well as an automatic music video emotion classification method in this work.In the case of music video emotion analysis, supervised methods have been conducted using the conventional approachA large dataset with precise annotation has become increasingly important for training the huge number of parameters of a data-hungry deep learning model. However, the data annotation process is relatively difficult and costly in the field of affective computing. For music videos in particular, the task is more challenging because it includes multiple information sources with their own emotion representation paradigm. The annotation on music videos is challenging even for an expert because emotion itself is a subjective task, dynamic across time, requires different emotion representation schemes for individual music video components , and is influenced by culture. Therefore, data scarcity appears if the affective computing algorithms are data driven. The unsupervised representation helps in the data-driven method to minimize data scarcity and annotation difficulties. The unsupervised network can use a massive amount of raw data floating around the Internet and expanding day by day. The classification of human emotions included in a music videos without annotation is most economic approach in the case like music video where the labeling task is expensive and complicated.To train the unsupervised neural network, we collected a large number of music videos from YouTube. A 30-s video clip was selected from each music video. The video clips were cut from a random temporal location of full length music video and downloaded using an automated python code. We set a search key related to music videos and downloaded both official and user-generated music videos. We filtered the clips because some were not related to music video and some had only music . The data was filtered to exclude reviews, interviews, and other music-related speech. Finally, our sample contained 0.13 million music videos for unsupervised training. The collected data were further processed using audio noise reduction techniques.50 follow the categorical model51, providing several discrete categories of emotions, and other datasets53 use the dimensional model54 to represent emotion as a value in 2D valence and arousal space. Similarly, like music, video emotion datasets56 have been proposed using the categorical model and others58 have used dimensional models.Several datasets have previously been proposed for supervised music emotion analysis. Some datasets59 was the first music video dataset available with 120 data samples taken from western music. The dimensional emotion representation model was used to rate affective tags in terms of arousal, valence, and dominance, using value ranges from 1 to 9. The dataset does not include enough samples to adopt the data-driven algorithms like deep neural networks. To overcome the data scarcity problem, we prepared a music video emotion dataset using the contribution of four annotators. The dataset is an extension of5 with the same emotion representation framework of six emotion categories, namely: Excited, Fear, Neutral, Relaxation, Sad, and Tension. The modified music video dataset included training, validation, and test samples of 4788, 655, and 300 video samples respectively. The test samples were equally distributed over the six class categories for a fair comparison. This new music video emotion dataset (MVED) has been used in36 and made publicly available.DEAP datasetWe categorized the dataset into six distinct classes based on their corresponding emotional adjectives. The “Excited” class usually includes positive emotions. The visual elements of the “Excited” class includes images of a smile on a face, movement of arms, dancing scenes, high lighting, and coloring effects. The audio components of this class include high pitch, large pitch variation, uniform harmony, high volume, low spectral repetition, and diverse articulations, ornamentation, and vibrato. The visual features of the “Fear” class reflect negative emotions via a dark background, unusual appearance, wide eyes, open mouth, a visible pulse on the neck, elbows directed inward, and crossed arms. Common visual elements in the “Tension” class are fast-changing visual scenes; crowded scenes; people facing each other; aggressive facial expressions with large eyes and open mouths; and fast limb movement. The audio elements in the “Tension” and “Fear” classes include high pitch, high tempo, and high rhythmic variation, high volume, and a dissonant complex harmony. The visual elements in the “Sad” class are closed arms, a face buried in one’s hands, hands touching the head, tears in eyes, a single person in a scene, a dark background, and slow-changing scenes. The “Relaxation” class includes ethnic music and is visually represented with natural scenes in slow motion, and single-person performances with musical instruments. The acoustic components of the “Sad” and “Relaxation” class include slow tempo, uniform harmonics, soft music, and low volume. The “Natural” class includes mixed characteristics from all the other five classes. The data samples in each class are diverse in terms of musical group, culture, nationality, language, number of music sources in one audio, and mood.62 have demonstrated that phase information improves the performance of both speech and music processing.The raw music video data needed to be processed in an acceptable form for the neural network. Our dataset processing follows several steps for each individual data sample of music and video. The music network is trained on the real (magnitude) and imaginary (phase angle) components of the log magnitude spectrogram. The magnitude of log Mel spectrogram was kept in one channel and the phase angle representation was placed on another channel to preserve both the magnitude and phase information of the acoustic signal. The raw music video data needed to be processed in an acceptable form for the neural network. Our dataset processing followed several steps for each data sample of music and video. The music network is trained on the real (magnitude) and imaginary (phase angle) components of the log magnitude spectrogram. The magnitude of log Mel spectrogram was kept in one channel and the phase angle representation was placed on another channel to preserve both the magnitude and phase information of the acoustic signal. Several studiesi was converted to mono and then subsampled with a window size of 2048, a sampling rate of 22,050 Hz, and a time shift perimeter of 512 samples. The sampling rate varied for the slow–fast network where xi in the slow path had a sampling rate of 32 kHz and the xi in the fast path had a sampling rate of 8 kHz. The Fast Fourier Transform (FFT) was then applied for each window to transform the xi from time domain to a time–frequency (T–F) representation Xi. From the entire frequency spectrum, 128 non-linear Mel-scale were selected that matched the human auditory system. The use of the log Mel spectrogram has two benefit compared to wave form audio. First; it reduces the amount of data that need to process by the neural network and second; it is related to human auditory perception and instrument frequency range63.For this work, the 30-s audio waveform xi = {v23nThe sequence of images on the video were collected in a distributed manner to preserve the temporal information of the entire video sequence. Each video was converted into several frames sequences Vmentclass2pt{minimN = N = After the preprocessing, the input to the audio network was A64. The CNN-based autoencoder is one of the popular unsupervised feature representation method66. At the time of writing, we are unaware of any unsupervised music or music video emotion representation method using deep learning technology. In this study, an autoencoder was trained on the encoder-decoder paradigm, where the encoder network used two multimodal architectures with a dense residual block with a variety of convolutions. The decoder network is made simple and lightweight using 2D/3D convolution. We will discuss the detail encoder and decoder network after the various convolution filter used in this study.In this paper, we present multimodal learning using the unsupervised method. In multimodal representation, the complementary information provided by the different modalities is integrated to enhance the system capability. An autoencoder network is developed to represent the multimodal representation using audio and video information. An autoencoder is a generative model, which usually uses a latent representation bottleneck and use it to reconstruct input68 have included this complexity and, as a result, require a large dataset for successful training. In this paper, the complexity of 3D convolution was reduced using filer and channel separable convolution. The proposed convolution was an integrated form of channel separable convolution69 and (2 + 1)D convolution70. For the separable filter, the 3D convolution filter of size 71. The channel size was reduced to one in the sequential block of the dense residual network for the separable channel. A detailed representation of the proposed filter and channel separable convolution is illustrated in the right most configuration in Fig. In video processing, 3D convolution has been found to better capture spatial and motion information, but it exponentially increases the complexity of a system. Some popular 3D networkson69 and  + 1D conTwo network architectures were used for information encoding, namely the “music-video encoder network” and “separable slow–fast encoder network”. Each network included an MMTM block for information sharing and feature enhancement. The music-video network used conventional 2D/3D convolution, whereas the separable slow–fast network used filter and channel separable convolution.The music-video encoder network has two parallel branches for music and video processing with an MMTM for information exchange after each dense residual block, as shown in Fig. Inside each dense residual block, the max-pooling operation is used for dimension reduction in the two parallel branches. Each parallel branch is then added together and the result is smoothed with a convolutional layer. Finally, the output from each individual dense residual block of audio and video branch is passed to the MMTM block for information fusion, as shown in Fig. The 2D/3D convolution in the music-video network increases the complexity and it is impossible to add more audio or video branches for end-to-end training. We drastically reduced the network complexity using filter and channel separable convolution and trained a slow–fast encoder network with audio and video information. Slow–fast networks can be described as a single stream to capture both spatial and motion information. The slow path is designed to capture more static but semantic-rich information, whereas the fast path is tasked with capturing fast motion. We used slow–fast representation for both the audio and video networks with because both media are spatially meaningful over time. Both slow and fast network branches are trained in parallel with information sharing using MMTM after each dense residual block. The MMTM module helps to modulate the audio and video correspondence over time. The architectural detail of the slow music video emotion network is shown in Fig. The fast branch of the slow–fast network has a similar network structure as the slow branch. However, video frames (64 frames) and sampling rate (32 kHz) are four times larger than the slow branch. Another difference is the size of the filter in the convolutional layer inside each dense residual block. To capture the motion information of the video, we process the input with a large filter size and low feature dimension. The architectural detail of the fast branch of the network is shown in Fig. In both fast and slow branches, the 3D convolution is limited to only one layer after a dense residual block for smoothing the learnt features. The dense residual block uses only a 2D convolutional layer with dense connection to the input so the network parameters are relatively low and easy to train in an end-to-end fashion. The detailed view of each dense residual block with filter and channel separable convolution is shown in Fig. In an autoencoder, the latent representation of input is reconstructed as output using a decoder network. A decoder network generally has the same structure as an encoder in reverse order from latent representation to input but other structures are possible. The goal of the decoder network is the reconstruction of encoded data in the original input form. The two decoder networks in this study: the “music-video decoder network” and “separable slow–fast network”, are designed to be simple and lightweight to reduce the number of network parameters of the multimodal architecture.73. The detailed architecture of the video and audio decoder network with multiple auxiliary outputs is shown in Fig. The music-video decoder network is designed to reconstruct the audio and video sequence from the separate branches. A simple 2D/3D transpose convolutional layer with batch normalization and ReLU non-linear activation function is used to upsample the feature from the latent space. The kernel size and strides are adjusted separately for each transpose convolution layer to make the final feature size equal to the input. The final video size is the same as the input, with three RBG channels, and the audio size is the same as the input log Mel spectrogram with phase information. The proposed decoder network introduces four auxiliary outputs for the video network in the expanding pathway and three auxiliary outputs for the audio network in the expanding pathway with the intention of improving the gradient propagation and decreasing the probabilities of a vanishing gradient for the deep audio–video multimodal networks. The multiple auxiliary outputs work as a kind of deep supervision and minimize the overall loss functionsThe separable slow–fast decoder network has two audio network branches and two video network branches without information sharing (MMTM block) across the audio and video network. The decoder architecture for the audio and video network in each slow and fast branch reconstructs their respective input dimension from their latent representation. The dimension of latent space and output of each audio and video decoder for the slow and fast path is shown in Fig. 1, c2, c3, …, cm ∈ C, audio AN = N = i |} for each of the m classes. This probability can be parameterized using the proposed multimodal M which looks at the joint representation of log Mel-spectrogram of audio and video frames in the music video to predict: P{ci|} = M{),  scores. We visually interpreted the system performance using confusion metrics and the ROC curve. The dynamic perceived emotion was evaluated by computing the time-synchronized emotion of the music videos.The loss function used for unsupervised feature representation was the mean squared error (MSE). MSE loss calculates the square of the difference between actual ground truth and network predicted result and average it over all the training data. The unsupervised features were fine-tuned with labelled data of music video emotion. At the decision level, all the branch information is globally aggregated and computed for a class-wise probability. The Softmax function was used for supervised training in the final layer of the neural network, which maps the output nodes in a probability value range between 0 and 1. We use the categorical cross-entropy loss function for a one-hot vector target. This function computes the binary cross-entropy for each class separately and then sums them up for the complete loss.In this experiment, we compared the two proposed networks using the evaluation score and network complexity. The networks were compared with the result of the end-to-end training and the unsupervised representation. In both cases, as shown in Tables The unsupervised feature of using the separable slow–fast network improved classification accuracy by approximately seven percentages. The music-video network also improved with the unsupervised features, as illustrated in Table The statistical evaluation score was investigated using the confusion matrix and ROC curve. The confusion matrix visualizes the class-wise correlation of the classifier prediction. It counts the number of confused samples for each pair of classes. The ROC curve illustrates the performance of each classifier according to the various decision thresholds associated with it. The confusion matrix in Fig. The ROC curve for the two multimodal architectures trained in the labelled and unlabeled dataset is shown in Fig. 76 to be essential cross-culturally to promote a song or album. The number of views on a music video also support this claim. In this experiment, we evaluated the relationship of human emotion to music videos. We present the results of our best performing network on two music video with billions of views on YouTube at the time of writing. The prediction results of “Despacito” (https://www.youtube.com/watch?v=kJQP7kiw5Fk) (7.54B views at the time of writing) and “Gangnam Style” (https://www.youtube.com/watch?v=9bZkp7q19f0) (4.19B views at the time of writing) are shown in Fig. An emotional relationship between consumers and music videos has been shown inThe prediction result (in percentages) for each class is illustrated with individual curves depicted across 30 s increments. We used the Sigmoid activation function to find the correlation of our six emotion categories. The highest activated class is illustrated with the highest score and the second or third most probable classes are further down on the vertical axis. One video frame is illustrated at the bottom according to the time for illustration, but the musical components are also responsible for the final decision.This research analyzed music video emotion using unsupervised and supervised methods. The dataset and methods are novel so an exact comparison with previous studies is not possible. The past studies are based on supervised method and there is no public dataset or method that treats the music video in an unsupervised manner, as far as we know.77 have presented varying evaluation scores for varying emotion class categories as shown in Table 46 first proposed a Gaussian mixture model (GMM) using low-level music and video features. The dataset used in that research was limited to 120 samples with an 8-class category. Another study47 used audio and video features for music video emotion classification from a pre-trained network and an inbuilt library. The study used 18 class categories at a feature level and the dataset is not publicly available. A deep learning-based study (MM in Table 5 used 2D and 3D convolution methods for audio and video processing respectively and used transfer learning for music video emotion classification. The limitation of this study was the large complexity and that the test samples were taken from similare training samples. A recent study MVF36 use the same labelled dataset used in this study for music video affective computing using audio, video, and facial expression. This study shows the performance increment using multi-modal representation and illustrated the emotion classification score for each information source.The researches related to music emotionIn contrast to previous research, this research used unsupervised and supervised methods and datasets for music video emotion classification using end-to-end training. Our dataset is more diverse and improved in the sense that it can be used to train the deep neural network and is publicly available with the hope of attracting new researchers. Compared to past studies, this study proposed new methods and used various statistical and visual analyses to support the resulting scores. In Table Music video affective computing is highly valuable for personal use to industrial application but little work has been done in this area in the past. The reason might be the lack of labeled dataset and annotation difficulties. Some research has focused on music emotion analysis; however, limited attention has been given to music videos, even though they are a widely used content on the Internet today. In this research, we present an unsupervised encoder-decoder paradigm for music video emotion representation. The unsupervised and supervised networks are compared with end-to-end training on a novel music video dataset. The visual and acoustic emotional clues of the music video are used in the multimodal representation with run time feature sharing and are boosted using a multimodal transfer module. The proposed multimodal architectures were first designed using general 2D/3D convolution and later the network complexity was drastically reduced by channel and filter separable convolution. We compared multiple unsupervised and supervised multimodal architectures using different evaluation metrics and visual analysis. The proposed method was found to be better in the qualitative analyses compared to past research.There is great potential for future studies related to music videos. The robust network architecture, standard dataset, emotion representation framework, and new evaluation techniques are an important part of this research. Time synchronized emotion analysis and multimodal architecture using more information sources such as lyrics, user comments, and facial information can open new opportunities for this field. We have made our code and dataset publically available to attract new researchers and receive feedback for this study.