However, most existing methods are prone to ignore the problems of visual information redundancy and scene information omission due to the limitation of the sampling strategies. To address this problem, a semantic guidance network for video captioning is proposed. More specifically, a novel scene frame sampling strategy is first proposed to select key scene frames. Then, the vision transformer encoder is applied to learn visual and semantic information with a global view, alleviating information loss of modeling long-range dependencies caused in the encoder’s hidden layer. Finally, a non-parametric metric learning module is introduced to calculate the similarity value between the ground truth and the prediction result, and the model is optimized in an end-to-end manner. Experiments on the benchmark MSR-VTT and MSVD datasets show that the proposed method can effectively improve the description accuracy and generalization ability. For example, automatic video captioning greatly helps users quickly screen out target content on video platforms. With the growth of neural machine translation (NMT)6, existing video caption methods are based on an encoder-decoder framework with recursive neural network (RNN)7 and convolutional neural network (CNN)8. The methods mainly focus on two parts of the video subtitle architecture, namely video encoder and subtitle generation. For the video encoder, the CNN-based encoder acquires a set of consecutive or equally spaced sample frames of the video and generates visual representation as input vectors for the model decoder. In particular, attention mechanism9 and time object diagram10 are also used to enhance the representativeness of the video in the video encoder. The other one is caption generation, LSTM network is adopted as decoder11, and complex language rules are further explored to improve the generation ability for the model. The visual encoder features and previously predicted output words are used as the input for the new module by the RNN-based decoder and gradually predicts the next word. Finally, a complete video captioning statement is generated.Video captioning aims to describe every scene given the video using natural language, and it is one of the most challenging tasks in computer vision because it requires the association of the video to text. It has been widely used in video searching12, compared with natural language, it is more reasonable to divide video into scene-based information units to understand the information expressed in the video. Therefore, we use scenario-based information grouping to understand video. For example, a video is composed of five scenes of information. In each scene, the largest difference frames are obtained, and finally, the input frame set of visual feature extractors is formed.Different from image captioning merely needing to comprehend the static content of the single image and generate the description, video captioning can understand the context information of the video in time sequence, rather than focusing on a single frame. Here, we consider that a high similarity exists between adjacent frames in the video, continuous video frames usually do not provide independent information. Therefore, each frame is regarded as a separate information unit, which may be not be the best choice for understanding video content. As mentioned in literatureWe propose a key scene frame screening method based on multi-scale image similarity measurement. By comparing the similarity of each frame, the best keyframe expression of the video in each scene is learned.We introduce an image captioning model to learn the description of the keyframe, and word2vec is used to transform the generated image captions vector into video semantic features. Capitalizes on the image captioning model’s proficiency in generating informative and contextually relevant descriptions for static images, thereby extending its utility to the dynamic realm of video content.Encoder blocks of Vision Transformer are used as feature encoder of the proposed model to achieve long-distance modeling and the feature hard encoder of CNN is learned from a global perspective to reduce loss of the hidden layer information in the middle of the encoder.A non-parametric metric learning module is proposed to optimize model parameters using the ground truth and the prediction result, and the description ability of our model is further enhanced to dynamically adapt video semantic information.The contribution of this paper can be summarized as follows: Video captioning is a complex and challenging task as it contains two different modalities, CV (Computer Vision) and NLP . In this section, the relevant works to imaging and video captioning are summarized respectively.15 is a fundamental task in vision-language understanding. Previous works mainly focus on the template-based method17 which needs to set multiple filling templates in advance. The filling template forms an empty slot by object relationship and attribute tag, generating image captioning statements. The generative process has some obvious limitations. For example, the generated sentences are fixed due to the limitation of limited templates designed by hand, and the generated sentences do not perform well in terms of naturalness and diversity.Image captioning18 combines the traditional image feature extraction techniques with the bowed word packet model. They improve the representation of semantic information of images by using TF-IDF in NLP to weight words. Vinyals et al.19 has proposed a neural image caption (NIC) model. They introduced a novel approach to enhance the long-term memory capability of the model, which is to replace the RNN in the m-RNN model using a long short-term memory (LSTM) network. The approach significantly improves the generation of image captions. The above work uses an encoder-decoder structure. First, the image semantic feature information is extracted by CNN. Then, the image caption statements are generated after decoding by RNN. Due to the feature extraction ability and natural language generation ability of the deep neural network, the generated description sentence structure is reasonable, smooth, and natural. Later on, Fan et al.20 proposes a novel grounded attention feedback model for automatic sentence generation from images. The generated text is used to rectify interest regions of images. In this strategy, the limitation of the one-way attention mechanism is broken out. The image caption generation method based on deep learning does not depend on the template with a single language. It is vivid and easy to comprehend but also has better training generation effects and speed than the template method. Li et al.21 introduced a framework that integrates attribute-based representations and attention mechanisms to enhance the understanding of visual content and generate descriptive language. By incorporating attributes, this method aims to capture finer-grained details in images and improve the quality of generated language descriptions. This work advances the synergy between the fields of vision and language by exploring the interplay between attributes and attention mechanisms. On the other hand, Ning et al.22 proposed an approach that focuses on learning representations and maintaining semantic consistency between remote sensing images and voice descriptions. By ensuring that the learned representations capture meaningful and semantically coherent information, improved retrieval performance is expected. This work emphasizes the significance of semantic coherence in learned representations, contributing to the fields of remote sensing and cross-modal retrieval.Recently, the framework of the encoder-decoder has been widely used in image captioning based on deep learning. Specifically, CNNs are often used to encode image features. RNNs are used frequently in the extraction of image features and in the generation of text descriptions phase. In recent years, research on graphical multimodality has made great progress. For example, Hong et al.24 to define a sentence template with grammatical rules. The sentence is parsed into subject, verb, and object, each of which is consistent with the video content. If a template with a fixed syntactic structure is predefined, the generated sentence will be fixed and single. In recent years, CNNs and RNNs have been commendably applied to various tasks. As a result, sequence learning methods27 are extensively used to caption video tasks with flexible syntactic structures. In recent years, deep learning models including CV, multimedia information retrieval, video understanding, and other tasks have achieved many excellent results. There are some typical examples such as action recognition28, and video question answering systems29. Therefore, the index results of the video captioning method based on deep learning are also significantly improved. The encoder is utilized to learn the visual coding representation of video. The decoder structure of generating sentences is used to convert the learned video coding representation into natural language description text with a more flexible structure and more human understanding. Venugopalan et al.7 used the sequence-to-sequence model. The LSTM is used to encode visual information. And then the corresponding sentences are generated through the modified LSTM. Xu et al.30 design a 3D-CNN encoder, which is used to generate visual features from video time and regions. Nian et al.31 presents an innovative approach to video captioning that focuses on learning explicit video attributes from mid-level representations. By explicitly modeling video attributes, the method enhances the semantic richness of generated captions. The paper emphasizes the importance of capturing diverse video characteristics, such as object attributes and actions, to enrich the context of captions. Through an intricate interplay of attribute learning and caption generation, the proposed technique contributes to the enhancement of video caption quality, paving the way for more contextually relevant and informative descriptions. Chen et al.32 propose the TDConvED module, which makes clear the learning action according to the theme and video dynamics at the same time. Zheng et al.33 introduce the transformer structure in the video captioning task. This approach reduces the redundancy in the video representation. The above methods have made great progress in natural language generation, but most models need to input all video frames or sample a fixed number of video frames at equal intervals. Video is known to consist of multiple consecutive video frames containing highly repetitive information. Taking advantage of this feature, researchers have designed some methods to encode video into information units through video captioning, a practice that mimics the way humans understand video content. Tang et al.34 introduce a pioneering approach that leverages the CLIP (Contrastive Language-Image Pretraining) framework for video captioning tasks. By extending the capabilities of CLIP to handle video content, the authors propose a novel method for generating captions that succinctly describe the visual content within videos. The fusion of image and text understanding is harnessed to enhance the model’s comprehension of video semantics, resulting in improved caption quality. This work showcases the synergy between multimodal pretraining and video captioning, contributing to the advancement of captioning techniques by effectively incorporating a state-of-the-art image-text embedding framework. HRNE35 uses a hierarchical LSTM network. Specifically, the input video is first encoded into a two-level abstraction. Later, BAE36 makes the assumption that a video is a set of continuous events. Then, the HRNE model is improved by seeking out the video hierarchical structure. Pick net37 assumes that all frames sampled at equal intervals do not necessarily contain meaningful information, but only select frames with certain significance for video captioning for model training. The above method encodes the input video by merging or discarding intermediate frames. Therefore, these methods do not consider the captions being generated. The same video features are used throughout the decoding process after the input video has been encoded.The video captioning task is spatiotemporal dynamic, multi-modal, and content diverse. So much so that it is more complex compared to image captioning tasks. In the early stage, the work of video subtitle adopts the time-based methodTo alleviate this problem, a semantic guidance network for video captioning is proposed. First, a scene frame sampling method is proposed using the image similarity algorithm for the model to perform critical scene frame selection, and then a semantic guided learning video captioning network structure is implemented based on metric learning. Which, we measure the similarity of each frame by MS-SSIM algorithm, remove the very similar video frames and keep the frames with large differences as the visual representation information of the video. Then, 2D, as well as optical flow and 3D visual features of the scene frames, are extracted, image caption of the scene frames are performed based on a pre-trained image caption generation system on the COCO dataset, and the generated description statements are converted into vector representations as semantic information of the video by a word2vec model. Secondly, the Transformer Encoder Block is used to form a model structure encoder to learn the above visual and semantic information with a global view and to reduce the loss of information in the middle hidden layer of the encoder for modeling long-range dependencies while learning a mixed representation of local and global features in the shallow layer. Finally, the output results of the decoder and the ground truth are used for Non-parametric metric learning, and the model is optimized end-to-end by minimizing the difference between the generated description and the ground truth. Through extensive experiments on two benchmark datasets, MSR-VTT and MSVD, it is shown that the proposed model can effectively enhance the model description ability of accuracy and generalization.38 uniformly samples 30 frames from each video, and small video clips around the sample frames are provided for feature extraction by 2D-CNN and 3D-CNN. Simultaneously, appearance and motion features are spliced to form frame representation information which is sent into the encoder-decoder constituted by LSTM to generate the final caption. Specifically, continuous video frames usually contain very similar semantic information, which can be divided into different information units when humans understand video. However, most video caption methods encode video into an information unit and after that processing, may discard or aggregate video frames with repeated information. Our purpose is to calculate the similarity between each frame of video through the MS-SSIM algorithm, to eliminate highly similar video frames. The original semantic information of the video with as few frames as possible is retained.Using natural language to automatically describe video content is an important task, and it has become a significant challenge in recent years. However, in the feature extraction phase, most existing models also use a random sample or uniform sample, and then the video caption content is generated via a high dimensional feature encoder and decoder. For example, literatureN videos, i.e., S frames and M corresponding natural language description text T, where i.e., 39, split proportion of i.e., We define the dataset as S frames from a video are divided by using the FFmpeg, i.e., X frame with the largest difference is reserved as the adaptive sampling frame set F of each video by calculating the structural similarity between frames, i.e., X the frame is taken as the input of 2D and 3D CNN during the encoder stage, and then i.e., F are selected as the input of the static image caption model on COCO datasets, and 5 corresponding text descriptions are output. Finally, the corresponding description is mapped as W via word2vec model, i.e., Y, i.e., Most existing video captioning methods mainly use encoder-decoder neural network structures. The architecture has two parts: encoder & decoder. It has been proven to be an advanced technology of machine translation. In Fig. i-th feature vector, N is the number of patches in the feature map, i-th patch, In order to avoid the redundant information of visual features extracted by continuous frames, we adopt the MS-SSIM algorithm to realize adaptive sampling frame. It culls highly similar frames and better preserves the visual information of the video. The adaptive scene frame selector based on MS-SSIM greatly reduces redundant frames and computational costs and ensures that key information is not lost. A details explanation is given in Section “41, involve explicitly choosing keyframes during the video encoding process. A keyframe selection layer is introduced to enhance a regular encoder, predicting the informativeness of each segment based on its contextual representation. While the aforementioned approach has achieved relatively accurate keyframe selection capabilities, the network structure that predicts based on context introduces computational complexity, particularly for longer video sequences, which may lead to higher computational overhead. Therefore, we propose an adaptive scene frame selector in this paper. Firstly, we adopt the MS-SSIM algorithm to compare the multi-scale structural similarities of all frames in the video, which generates similarity scores between two frames as shown in Fig. Compared to images, video typically contains more redundant information in consecutive frames, which makes the model more challenging in terms of capacity and computational efficiency. The current mainstream models mainly focus on visual task processing and natural language generation tasks, neglecting the precritical frame selection, which may lead to the degradation of model caption performance. In the field of video keyframe selection, the current mainstream keyframe selection methodsFirstly, the video is segmented into several frames. An initial keyframe is selected and the MS-SSIM score between the initial keyframe and each frame in the video is computed. Then, a threshold is set based on the mean and standard deviation of the MS-SSIM scores. The remaining frames in the video are traversed, and the MS-SSIM score between the current frame and the previous key frame is calculated. If the MS-SSIM score exceeds the set threshold, the current frame is set as a new keyframe. Finally, these steps are repeated until all frames have been processed, and the final output is the key frame sequence. The algorithm is described in Algorithm 1. The original semantic information of the video is preserved in a few frames, and the sampled frames with richer semantic information are obtained to extract visual features.42 (MS-SSIM) is a widely used image quality evaluation metric. The method attempts to quantify the error between distorted images and reference images by using various known characteristics of the human visual system. As shown in Fig. The multi-scale structural similarity index metricFor images x & y, the SSIM is calculated as follows:L is the luminance contrast factor, C is the contrast factor, and S is the structure contrast factor.First, calculate the mean of the image:is the mean of x, is the mean of y, is the squared difference of x, is the squared difference of y, andis the covariance of x and y.SSIM calculation:SSIM satisfies the following properties.Symmetry: Boundedness: M-1 iterations is M. For the j-th scale, only the contrast c and the structural similarity s are calculated. Luminance similarity l is calculated only when the scale is M. Finally, the results of each scale are concatenated to obtain the MS-SSIM value, as shown in Eq. , so that the network has more nonlinearity and better fits complex correlations between channels, while preferably reducing the number of parameters and computational effort. The ResNeXt network is a combination of ResNet and Inception51. First, I3D pre-trains each structure on the Kinetics dataset and then adjusts slightly each structure on the datasets HMDB-5152 and UCF-10153 to enhance the pretraining performance of the I3D model and enables it to better adapt to downstream tasks.In processing video temporal motion information, we use the Two-stream Inflated 3D ConvNets (I3D) model pre-trained on the kinetics datasetFrame Semantics Feature The video is constructed from multiple static frames in a temporal sequence, so the information generated by describing the content of each scene keyframe selected through an adaptive frame selector in the video contributes to the overall video content description. The sampled frames are captioned using an image caption generation system43 pre-trained on the authoritative Microsoft Common Objects in Context (COCO) dataset, and then the corresponding subtitle text is converted to a 300-d vector representation using the word2vec model. The literature54 also states that Word2vec trains word vectors according to context information, so the word vectors trained by the word2vec model contain context semantic information.Vision Transformer Encoder Conventional encoder and decoder models are usually RNN structures, but there are two problems in the sequential calculation of RNN series models. a) Each time state is dependent on the previous time step state, which results in the model not being able to be accelerated by parallel computing. b) A series of models in the RNN series, such as GRU and LSTM, have introduced a gate mechanism. But they have the disadvantage of a limited ability to handle long-term dependency problems. Inspired by the success of Transformer55 on a wide range of tasks in NLP, Vision Transformer (ViT)56 is the first to adopt a pure transformer architecture for image classification, which shows the good performance of transformer architecture on vision tasks. Therefore, we also use the transformer encoder as our model encoder when designing the model structure, which can effectively use the intermediate state of the transformer encoder to achieve global encoding of video features.l-th time step shown by the encoder module structure in Fig. N features are normalized as the final feature representation of the encoder. The output size of this encoder is 1024-d. The calculation process is shown in Eqs.  and Multilayer Perceptron (MLP). The multi-head attention structure is shown in Fig.  in Eqs. –15.13\doLSTM, as a sequential model, progressively generates output sequences. In the context of video captioning, the generation of descriptive sentences typically involves a temporal sequentiality, where frames are generated in accordance with chronological order. LSTM demonstrates prowess in handling such sequential generation tasks. In contrast, Transformer’s self-attention mechanism may lead to heightened computational complexity during the decoding phase, particularly when generating longer sequences. In comparison, LSTM presents relatively diminished computational overhead, thereby rendering it more efficient in the decoding process. This efficiency assumes an advantageous role when grappling with extensive video datasets and lengthy sequences. Given the pronounced temporal nature of videos and the underlying sequential dependencies among corresponding natural language descriptors and words, we observe that our generated captions remain relatively succinct, obviating the necessity for handling extensive dependencies. Consequently, the decoder module of our model adopts a multi-layered architecture of Long Short-Term Memory (LSTM) networks, which serve as the foundational structures for the decoding process. The LSTM unit network structure is shown in Fig. t, it is assumed that the model encoding output is a 1024-d vector At time  in Eqs. –21.16\doWe obtain the corresponding hidden layer output In this section, nonparametric metric learning is used to reduce the discrepancy between model-generated and ground-truth captions. By comparing the captions generated by the model with the ground truth, the vector representations of the generated captions are semantically similar to the ground truth.57 is utilized to compare two 1024d vectors There are two sequences in the metric learning module: the video captioning sequence generated by the model and the real data video captioning sequence. We employ an LSTM network structure to embed sentence sequences into a sentence-vector model. Each of these LSTMs has 1024d hidden and output layers. Furthermore, a cosine similarity lossd as Eq. .22\documL as shown in Eq.  is presented based on the joint presentation of the MPII Movie Description Dataset (MPII-MD) and the Montreal Video Annotation Dataset (MVAD). The dataset consists of 128k sentence fragment pairs and 158h of the video. The training, validation, public, and blind test sets contain 100000, 1000, 1000, and 1000 video clips, respectively. Among them, the blind test set does not give the standard description sentence in advance, so the results need to be submitted to the CodaLab website, and the website will give the evaluation score.60, METEOR61 ROUGE-L62 and BLEU 63 are used as evaluation metric to calculate evaluation scores for our model and comparison model, to objectively evaluate the statement description generation effect of the final model.We adopt four algorithms widely used in the video captioning field, CIDErVocabulary Preprocessing For each sentence, we first segment the sentence using the Stanford NLP Toolkit(https://stanfordnlp.github.io/CoreNLP/) and truncate it into 20 words, and then remove punctuation and convert each word to lowercase. In order to form vocabularies, the minimum occurrence frequency of each word is set as 1, which ultimately resulted in a vocabulary of 7,184 words for the MSVD and 16,860 words for the MSRVTT. To specific, random initialization is used to embed words.Feature Extraction When extracting features, each frame is first randomly rotated by Encoder-decoder Before inputting the video features into the model, the visual features and semantic features are mapped into 1024-d vectors respectively, and the words are embedded into 300-d vectors.T denotes the total number of frames in the video. For each image t by concatenating them along the channel dimension, C is the total number of channels in all the features.t-th position in the sequence. The Vision Transformer consists of multiple Transformer encoding blocks and uses the default implementation.In the feature extractor, firstly, we represent the video as a sequence of images n in Eq. . We stacn in Eq. . Finallylows Eq. .24\docum64 is experimentally applied to optimize the proposed model, and the initial value of this optimizer parameter is 65 is used in this paper to achieve regularization. The initial values of all the weights to be gradient updated are set to be uniformly distributed on the interval . The set search size for the model testing phase is set as 5.In the model training phase, the AdamCross-entropy loss In this paper, the cross-entropy loss function is applied to calculate the deviation of the target value from the actual output value. The optimal value intervals used for positive and negative samples are obtained through extensive experiments. This function turns the output of the neural network into a probability map. From this, the cross-entropy can be used to compute the distance between the predicted probability map and the actual output value. Meanwhile, the inclusion of the equilibrium parameter x presents the fully connected result, y indicates the ground truth, and P is the softmax. in Eqs. –28.27\doWhen incorporating the balance parameter into the cross-entropy loss function, the impact of different types of samples can be adjusted by assigning different weights to positive and negative samples. Suppose there are n samples, where the true label and predicted probability of the i-th sample are denoted as d as Eq. .29\docum66, SGN38, SCST67, STG68, SAAT33, E2E69, and POS-CG70 are used as competitor models. The qualitative results can be found in Tables To further verify the performance of the proposed method, it is evaluated on the MSVD, MSR-VTT, and LSMDC datasets, respectively. VC-HRNATIt can be seen from Table Furthermore, we further conduct and analyze comparative experiments on the MSVD benchmark dataset to validate the performance of our method. The detailed results can be found in Table Table From Tables The above results suggest that our model not only performs well on the mainstream MSR-VTT and MSVD datasets but also achieves good performance on the practical application dataset LSMDC, which further verifies the generalization performance of the model in this paper. In addition, it can be seen from Fig. The performance of the video content understanding model relies on the backbone of CNN. In order to verify the effectiveness of different parts in our model, the performance of the model is compared in different scenarios by modifying or removing a part of the model, or changing the training strategy. In the two different cases where the encoder uses the conventional LSTM and the Encoder Block of the Vision Transformer architecture, six different backbone combination architectures are performed separately for model performance testing, namely VGG, ResNet152, SE_ResNet152, VGG combines with ResNeXt101, ResNet152 combines with ResNeXt101, SE_ResNet152 combines with ResNeXt101, and the 3D, as well as optical flow features, are extracted by choosing the I3D structure fixed.The difference between using and not using metric learning in model training lies in the optimization objective of the model during training. In traditional model training without metric learning, the objective is typically to minimize a loss function, such as cross-entropy loss. In this case, the model is trained to directly predict the class or probability distribution of the samples. In contrast, in metric learning-based model training, the objective is to learn a distance or similarity function that can map similar samples to nearby feature space. In this case, the model is trained to predict the similarity between samples, rather than directly predicting their class or probability distribution. This approach is typically used for learning low-dimensional embeddings for easier data visualization and analysis.Analyzing the results of ablation experiments, it can be observed that models trained using metric learning can have better generalization performance, as they consider the similarity between samples rather than just their class labels. Additionally, metric learning can handle data in non-Euclidean spaces, such as text and images.To better quantify the impact of different backbones on our model performance, several ablation studies are conducted to investigate the performance variation due to different backbone structures. It is noticed that each backbone ablation model refers to fixing other components and changing only the feature extraction module, and all ablation experiments are performed on the same dataset segmentation and training settings, and the results of the MSR-VTT benchmark dataset experiments are shown in Tables The SE-ResNet152 introduces a channel attention mechanism called Squeeze-and-Excitation, which improves the model’s focus on important features by learning the importance weights of each channel. On the other hand, ResNeXt101 uses group convolution to increase the model’s width without increasing its computational complexity, which enhances the network’s expressive power. In this work, we propose a model that combines the features extracted from both SE-ResNet152 and ResNeXt101 as 2D feature representations. Our experiments show that compared to using a single 2D CNN, the training time of the proposed model increases by approximately 10% to 12% due to the use of fused features.Effect of encoder on model performance Current model frameworks for deep learning video caption generation are typically designed as encoder-decoder architectures, in which the encoder learns a compressed representation of the video from multimodal features, and the decoder uses the video representation output from the encoder to generate a sentence word-by-word. The learning of video encoder representations is the basis of the video understanding, so the quantitative results of the ablation experiments with different encoder architectures while fixing other model structures are shown in Tables Impact of metric learning modules on the model structure In order to enhance the accuracy of the proposed model, a metric learning method based on cosine similarity is introduced. The quantitative results of its metric validity test are shown in Table We propose a novel semantic guidance network for video captioning, alleviating existing methods that cannot make good use of the visual representation and long-distance encoding. Our model consists of a scene frame selection module, visual feature and semantic information extraction module, ViT encoder block encoder, and LSTM decoder module. Specifically, the proposed model can make full capture of representation information of each scene and achieve the longer-distance dependencies encode representation. In addition, a non-parametric metric learning module is used to further optimize the description performance of the proposed model. Extensive experiments verify the description effectiveness of the proposed method on two mainstream datasets MSR-VTT and MSVD. Furthermore, the description performance is further verified on the real-world engineering application datasets LSMDC.In future works, we will focus on the following directions. First, novel attention mechanisms and reinforcement learning methods can be used to further improve the descriptive performance of the proposed model. Second, the domain adaptation strategy seems a better choice for video captioning.


Objective structured clinical examination (OSCE) results could be affected by low homogeneity of examiners, non-retrospectiveness of test results, and examiner-cohort effect. In China, many students participate in medical qualification examinations, and this issue is particularly significant. This study aimed to develop a video recording, video-based rating method and compare the reliability of video and on-site ratings to enhance the quality assurance of OSCEs.The subjects of this study were clinical students one year after graduation participating in the clinical skills portion of the National Medical Licensing Examination. The participants were from four cities in Jiangsu province. Participants were randomly allocated to on-site and video rating groups to evaluate the rating methods consistency. We verified the reliability of recording equipment and evaluability of video recording. Moreover, we compared the consistency and equivalence of the two rating methods and analyzed the impact of video recording on scores.The reliability of recording equipment and evaluability of video recording were high. Evaluation consistency between experts and examiners was acceptable, and there was no difference in evaluation results (P = 0.61). There was good consistency between video and on-site rating; however, a difference between the two rating methods was detected. The scores of video-based rating group students were lower than those of all students (P < 0.00).Video-based rating could be reliable and offer advantages over on-site rating. The video recording, video-based rating method could provide greater content validity based on its traceability and the ability to view details. Video recording, video-based rating offers a promising mthod for improving the effectiveness and fairness of OSCEs. The assessment of clinical competence within medical education primarily occurs through written examinations designed to evaluate medical knowledge and performance assessments, such as the objective structured clinical examination (OSCE) to evaluate clinical skills ,2. OSCEsStudies have demonstrated multiple threats to score reliability due to assessment differences , with thIn the physical examination and essential operation skills test stations in China, examinees perform corresponding operational checks, and examiners perform real-time scoring based on the scoring standards and the students’ actual operations. The operation process of students according to the requirements of the test questions is a continuous series of actions. The examiner carefully observes the students’ performance, judges whether the students’ operations are standardized, and then scores based on the grading rules. If the examiner misses or fails to see clearly due to subjective, carelessness or lack of responsibility, or objective factors, such as long work cycle and high intensity, points may be missed or deducted , threateExaminer training is considered a means to improve test evaluation reliability; however, there may be inconsistencies in the effectiveness of such training ,12. As tVideos have been used in all aspects of medical education. Since the 1980s, videos have been used to assess communication skills  and wereAlthough video recording has theoretical advantages, in practical applications, the video recording system must accurately capture the details of the operation to prevent students’ bodies or heads from inadvertently blocking the required vital angles. In addition, there are high requirements for the ability of cameras and equipment to achieve high audio and video quality. Furthermore, whether video rating between raters is as reliable as the on-site rating remains unclear.This study recorded examinees’ practical skills assessment process using the video recording, video-based scoring method and adopted multi-camera synchronization recording to form a video answer sheet for the examinees’ practical skills evaluation. The examiners use the network scoring terminal (referred to as the software platform) to assess videos on three aspects: consistency of evaluation methods, difference in evaluation results, and impact of the video recording system on student scores. This study posed the following questions: (1) Can video recording equipment function reliably? (2) Are scores different between video and on-site ratings? (3) Does the presence of video cameras in the room impact scores? This study aimed to develop a video recording, video-based rating methodology and offer a promising framework for improving the effectiveness and fairness of OSCEs.In 2020, 4,714 students participating in the clinical skills portion of the National Medical Licensing Examination from four cities in the Jiangsu province  were selected. All procedures performed in this study were approved by the ethics committee of Nanjing Medical University and all participants agreed to participate in the study.The clinical skills portion of the National Medical Licensing Examination consisted of three assessment components, with the first station evaluating clinical thinking. The test questions were delivered by a computer. The test duration of the first station was 40 minutes. The second station evaluated physical examination skills, with a test during of 15 minutes. The third station contained 24 questions on basic clinical skills, including skin disinfection and draping, suturing, abscess incision and drainage, cardiopulmonary resuscitation, etc. The test time was 10 minutes. Candidates completed one question from the second and third station for examination. The physical examination required cooperation of a simulated patient, and basic clinical skill operation was conducted on a simulator or a mold.The audio and video recording equipment set consisted of a six-camera surrounding unit, a three-camera independent pickup, and a recording and broadcasting workstation to ensure the critical operations were not obscured. During the exam, the audio and video recording equipment documented the complete process, which was integrated into a format with six channels of video and three channels of audio as H.264 format files . The hash value, examinee, examination group, and examination time were calculated, and files were uploaded to the base server. Video answer files were produced. Backup files were stored on the workstation.All examiners were experienced clinicians. To improve the homogeneity level of examiners and the efficiency of scoring, the video-based score adopted a classified scoring method, and each examiner only evaluated one type of test. After intensive training, classified training, and test evaluation, the examiners assessed on-site or via video answer files in Nanjing Medical University. Examiners who assessed via video answer files should send their evaluation within three days.After examination, video answer files were sent to examiners for scoring through the management system network. Each examiner was only assigned one type of exam question. The scoring system provided video viewing angle and audio position switching functions to facilitate examiners to change the audiovisual position. Audio and video could be synchronized fast forward, slow forward, playback, and zoom in any position, and the examiner recorded the scoring results on the structured scoring table. Two examiners independently evaluated each video answer file, and the system automatically submitted the assessment difference to the chief examiner for confirmation to determine whether the evaluation difference exceeded the threshold. The process is represented in a flow chart .Figure 1. Video-based scoring examiner selection and video answer file samplingAs the experimental group, three groups were selected from the second and third stations. The surround recording and broadcasting equipment recorded all students in the experimental group to form a video answer file. Moreover, the examiner was assigned to conduct on-site scoring. In the experimental group, each group selected an examiner to participate in the video-based scoring. The video-based evaluation was performed on 30 randomly selected examination video answer file that the examiner assessed on-site.2. Sample answer file selection and assessmentNine typical video answer files were selected from the second and third stations’ video answer files as reference samples. The test experts and examiners chosen from the second and third stations conducted video-scoring of reference samples.We counted the number of video answer files that should be received and the number of video answer files actually received in the four bases. Equipment reliability was measured as the number of video answer files actually received divided by the number of video answer files receivable. The examiners evaluated the quality of audio and video recordings of all scoring points during network rating. Evaluability was measured as the number of evaluable scoring points divided by the total number of scoring points.P < 0.05, where the adjustment was made according to multi-comparison correction.Measurement data were expressed as the mean ± standard error of the mean (SEM). The non-parametric Wilcoxon rank-sum test evaluated the difference in scores between the two groups with and without video recording in the second and third stations and the difference in video scores between experts and examiners in the second test station. The intra-group correlation coefficient (ICC) was used to evaluate the consistency of the scores of the two video examiners at each station, and the F test was used to test the hypothesis of ICC. ICC is equal to individual variation divided by the total variation with a value belonging to , where 0 indicates untrustworthy, and 1 indicates completely trustworthy. It is generally believed that a reliability coefficient lower than 0.4 indicates poor reliability, and greater than 0.75 indicates excellent reliability . All tesIn 2020, the expected number of videos received by candidates from Nanjing, Suzhou, Nantong, and Xuzhou was 1909, 1008, 887, and 910, respectively. The actual number of videos received was 1909, 1008, 887, and 910, respectively, and the missing rate was 0, indicating that the video recording equipment was reliable.There were 505 scoring points for expert evaluation, 494 scoring points for evaluable points, and the evaluability given by experts was 97.8%. Examiners’ evaluations included 23,107 items, evaluable scores included 22,967 items, and evaluability given by examiners was 99.4%. These results demonstrated that the examiners and experts gave high evaluability to the video record.P = 0.61; A comparative analysis of the differences between the examiner and expert video scoring results revealed that the average score of examiners was 14.45, and the SD was 2.95. The average score of experts was 14.82, and the SD was 2.60. There was no statistical difference between the results of the examiner and experts for the 18 reference samples (P < 0.001) and 0.87  and 0.87 (P = 0.004), which indicated high reliability in examiners.Based on 9 sample video observations and scoring, the consistency of the average scores of the 12 examiners and expert scores at the second and third stations was evaluated. The results showed that the average scores of the examiners at the two stations and the expert scores were highly consistent. The ICC was 0.94 (P = 0.046). The results of the third station showed that the video rating score  was higher than the on-site rating score , which was lower than the on-site rating score  and 13.41 (SD = 3.99) on the second and third test station, respectively, which were both higher than the scores of video recording group (mean = 14.61, SD = 2.99, mean = 12.49, SD = 4.24; The clinical skill test is an integral part of the physician qualification exam in China. At present, the clinical skill test mainly takes the form of on-site examiner scoring. However, the traditional examiner grading model has certain shortcomings. The test process cannot be reproduced, and test scores cannot be traced back and verified. Some examiners are not strictly controlled, which could lead to undesirable validity and consistency. The organization of assessments requires a large number of experienced physicians to serve as examiners, which brings considerable pressure to the clinical work of local hospitals. Clinical skills test uses a multi-channel OSCE test station, which makes it difficult to achieve homogeneity of evaluation. Examiners in the same test group have to evaluate multiple examination questions, which causes the examiner to be more stressed.Video rating is a digital examination evaluation mode that records the student’s performance process in a clinical skill test. This progress is then played back through network and multimedia equipment, and the examiner performs comprehensive scoring according to structured scoring standards. The aim of video rating is to subjectively eliminate the examination view of emphasizing results and neglecting process, establish an examination view that emphasizes both results and process, and appropriately highlight process. During our study, we found that video-based rating offers potential advantages in OSCE. One of these advantages is that the recorded video can be used as effective learning resource for students to intuitively understand the problems that occur during OSCE. Another advantage of video rating is quality control. Due to traceability and detailed inspection, examiners can assess exam skills according to standardized grading criteria, which can improve the objectivity of assessments.One of the most important factors of OSCEs scoring is reliability. Vivekananda-Schmidt et al. investigated the inter-rater reliability of OSCE stations for shoulder and knee examinations of 95 third-year medical students between real-time and video-recorded evaluations and found that the scores may not be equivalent . Sturpe When designing video recording, vital issues, such as necessary equipment and professional knowledge, need to be considered to obtain high-quality audio and video data. Nanjing Medical University has continuously improved the audio and video collection programs and technologies. This study used a 360-degree surround video camera supplemented by three independent pickups to collect stable and clear pictures, high-quality sounds, and wide angles to meet the needs of examination evaluation. In addition, video recording could be an essential reference material for students to improve their operational skills. However, the performance of students may be affected by the video recording process . This stFurthermore, this study identified additional advantages of video rating. Examiner evaluations and student examinations did not need to be performed simultaneously, increasing flexibility in scheduling and reducing time constraints. The efficiency of video-based rating was also higher than that of on-site rating, alleviating pressure on examiners. Each examiner was assigned only one type of examination question via video-based rating method, significantly reducing the pressure of scoring and improving assessment efficiency. Additionally, each examination question was independently evaluated by two examiners, and the system automatically submitted any differences exceeding the threshold to the chief examiner for confirmation evaluation, enhancing evaluation homogeneity.Although our study showed that video recording and video-based rating had significant strengths in OSCEs, they nonetheless have some limitations. We conducted a statistical analysis to determine whether video recording had an impact on the scores. Although we did not find a significant effect on the scores, further research is needed to improve this conclusion. This could include questionnaire analysis of participants to determine whether the video recording affected their performance. Additionally, the examiner’s feedback on the experience of using the video evaluation, including operability, video clarity, and evaluability, are also important indicators for evaluating the application of the video recording and video-based rating method.Future studies should address the limitations described. The video recording process needs further optimization to ensure the clarity and evaluability of the recording details, providing high-quality video information to the examiner. Studies should explore the perceptions of examiners and participants on the use of video recording, video-based rating method, and the impact of interventions on assessment behavior. Further research should be conducted on the impact of video recording feedback on teaching levels and candidates’ ability improvement. Additionally, more studies on the objectivity and evaluability of video-based rating methods need to be explored and refined.The traditional scoring model is not sufficiently objective and efficient. This study revealed that the video rating mode could eliminate the examiner’s intervention with the students on-site and improve the objectivity and fairness of examinations. Moreover, the method of categorized scoring could reduce examiners’ work pressure, improve examiners’ efficiency, and evaluation homogeneity, ensuring an objective and effective evaluation of students’ clinical skills. Therefore, it is necessary to extend this model to practicing physician qualification examinations.


Video gaming and competitive gaming (esports) are gaining more and more recognition in society as well as in research. Increasingly, health-related topics are the focus of research on video game and esports players. Although video gaming is often associated with energy drinks and fast food, no studies have yet examined the players’ dietary behavior. Therefore, the aim of this cross-sectional study is to investigate the dietary behavior and additional health-related data of video game players and esports players in Germany.Between July and October 2020, 817 participants , divided into video game players and esports players, were surveyed via an online questionnaire about their dietary, health, and gaming behaviors. Descriptive statistics were performed on all questions. To investigate statistically significant differences between video game players and esports players, the Mann–Whitney-U-Test and Kruskall-Wallis-Test were used. Partial Spearman correlations were used to examine possible associations between dietary behavior, health status, well-being, and video game playing time.Water was the primary source of fluid intake for the players (10.9 ± 7.0 l/week). The average weekly consumption of energy drinks was 0.4 ± 0.9 L. Energy drinks  as well as soft drinks  are positively correlated with the video game playing time. Participants ate 7.5 ± 10.4 servings of fast food per month, which has a positive association with video game playing time . In contrast, vegetables (1.7 ± 1.6 servings/day) and fruits (0.9 ± 1.0 servings/day) are eaten almost daily.In this survey, the dietary behavior of video game players and esports players is similar to that of the German general population. Nevertheless, there is a need for improvement. Especially energy drinks, which are already documented to have adverse health effects, should be limited. In addition, the consumption of fast food and meat should also be reduced, and healthier foods such as fruits and vegetables should be increased instead. Early education and support regarding the associated risks with unhealthy foods is important within the target group. Competitive gaming (esports) and video gaming have long ceased to be a simple trend. In recent years, the field has grown steadily, gained more and more recognition in society, and is becoming increasingly popular as a recreational activity. The numbers of active players and people who watch video games via streams are rising every year . The COVThe rising interest in gaming is also reflected in research. Since 2015, publications have grown consistently . ScientiHowever, there is a lack of evidence when examining the diet of gamers. Hence, the aim of this exploratory study is to investigate the dietary behavior of video game players and esports players in Germany. Additionally, health behaviors and body image were surveyed. Collecting data on the players’ dietary behavior is essential for providing specific health promotion to this at-risk target group over the long term.esportwissen.de), and social media  were used. In addition, esports organizations were contacted, and gaming websites wrote articles about the survey to further promote it. To motivate potential participants for the survey, the aim of the study and eligibility criteria were explained during the distribution. For an additional incentive to participate in the survey, participants were able to enter a voluntary raffle after completion . The study size was based on the previous studies and aimed to reach around 1000 participants during the survey period. The ethical committee of the German Sport University Cologne approved the study (reference: 053/2018).The current study is a successor to the eSports Study 2019  and 2020Participants were eligible if they lived in Germany at the time and understood the German language in order to reach only the German gaming scene. Other participation requirements included being 14 years old or older and actively playing video games. The first page of the survey showed an information sheet specifying the purpose of the study, and consent to participate was obtained. Before the actual survey started, the participants were asked about their country of residence and age. For participants who did not live in Germany or who were younger than 14 years, the survey ended, and they were excluded.Questback GmbH, Cologne, Germany).The questionnaire was designed to examine the dietary behavior of video game players and esports players. Demography, video gaming behavior, and health behavior were also included as topics. Overall, the questionnaire contained a total of 61–63 questions, depending on the participants’ answers to the filter questions. It was created and conducted using the online survey tool Unipark ,former professionals ,amateurs ,regular players ,occasional players ,non-players.Moreover, participants should indicate their favorite game genre (a semi-open-ended question) and their video game playing times (hours per week). The game time was divided into four categories:alone against human opponents (player vs. player),with human players together against human opponents (koop-player vs. player),alone against computer-controlled opponents (player vs. environment),with human players together against computer-controlled opponents (koop-player vs. environment).The questions on the health behaviors of the participants referred to their last four weeks. The self-rated health was measured with the first question of the SF-36 questionnaire [WHO-Five Well-Being Index [Subsequently, gaming behavior was investigated with self-designed questions, because no appropriate and validated questionnaires were available. First, participants had to classify their player status:ionnaire . The durng Index  was implIf they are vegan or vegetarian,Which foods they avoid .The dietary behavior was surveyed according to the food frequency questionnaire of the Robert-Koch Institute [The main part was introduced with two questions referring to the diet of the participants:nstitute . It can The last part of the questionnaire consisted of the participants’ personal body image and their desired body image. For this purpose, the body image scale of Stunkard, Sørensen, and Schulsinger 1983) was used  or implausible . Participants were also excluded if they did not complete the full survey. In the case of missing data for individual questions, the participant was excluded only for the corresponding parameter. For calculations, list wise exclusion was chosen to maintain a consistent number of subjects in the tables.For a clearer analysis of the dietary data and more clearly arranged tables, the player groups were grouped into video game players and esports players. The esports players include professionals and amateurs. All other groups were summarized in the video game group. The term “gamers” is used for both groups together. Furthermore, for clarity, the foods related to dietary behavior were grouped together in the tables.The participants' weight and height were measured, and their body mass index (BMI) was calculated. The playing times of the individual game categories were added together to obtain the total playing time. The amount of drinking and food consumption was recorded monthly, which corresponds to 28 days. For daily consumption, the servings were calculated down. For the calculations of the correlations with body image, first the difference between the desired body image and the current body image was calculated. Based on this, the differences were categorized from 1 = agreement to 9 = greatest difference.Descriptive statistics were applied to all the data. For each player status, numerical data as well as ordinal data were described by means, standard deviations, median and IQR. Nominal data were described by frequencies. All data were checked for normal distribution. If a non-normal distribution was present, statistical differences between the groups were calculated by the Kruskal–Wallis test, Fisher’s exact test, or Mann–Whitney U-test. Otherwise, ANOVA was performed. Post-hoc tests with Bonferroni correction were performed if overall group differences were statistically significant. Sign test was used to calculate differences between the current body image and the desired body image. To examine possible associations, partial Spearman correlations were calculated. The resulting coefficients were interpreted according to Cohen  as poor In Fig. 2 (n = 683), which can be classified as overweight. The percentage of female gamers differs between video game players (15.7%) and esports players . The Kruskal–Wallis test showed differences in education across groups (p = 0.04). However, according to the post hoc test with adjusted significance, no differences were found between the individual groups. There are no more significant differences between the video game players and esports players based on the sample characteristics.Table Table In comparison between the groups, esports players (8.25 ± 3.6 h/day) sit statistically significantly longer than video game players . Additionally, the esports players (8.3 ± 8.0 h/week) are significantly less physically active than the video game players . Regarding the playing time, all player categories playing statistically significant more than the occasional players (p < 0.01). Furthermore, the amateurs (26.7 ± 14.9 h/week) play more hours in a week than the regular players do .Table Except for energy drinks, no statistically significant differences in drinking consumption were found between video game players and esports players. Esports players (0.2 ± 0.6 servings/day) drink significantly more energy drinks than the video game players .Most of the participants (85.2%) do not follow a specific dietary pattern. The other participants (14.8%) are vegetarians or vegans. Overall, nearly half of the gamers prepare their own meals at least five times a week. No statistically significant differences were found between the video game players and esports players.Table The results show that both vegetables (1.7 ± 1.6 servings/day) and fruits (0.9 ± 1.0 servings/day) are consumed almost daily. For these foods, there are no significant differences between the groups.To have a better idea of the serving amounts, animal-based products are given in servings per month here. It shows that the esports players (16.4 ± 18.5 servings/month) eat more red meat than the video game players . Additionally, esports players consume 12.8 ± 16.6 servings of poultry a month, while video game players eat 9.7 ± 13.7 servings a month (p < 0.01). In contrast, video game players consume statistically significantly more dairy products and cheese than esports players.The esports players (9.1 ± 11.6 servings/month) consume statistically significantly more fast food than the video game players . However, the video game players (18.2 ± 26.5 servings/month) eat statistically significantly more sweets than the esports players . The same applies to sweet bakery products.Table There are no significant differences between video game players and esports players (p = 0.56). Even if the groups are divided by gender, there are no significant differences between them .Table The findings show a positive but weak relationship between playing time and sedentary time per day . There are also poor positive correlations between playing time and energy drinks  as well as fast food . Apart from the correlations shown in the table, there are no other statistically significant correlations.Table The aim of the eSports Study 2021 was to investigate the health and dietary behavior of video game players and esports players in Germany. Data from over 800 mostly male, young, and well-educated participants show that self-reported health is generally good. While the players have relatively high sitting times, the majority are also physically active. Nevertheless, adults are slightly overweight on average. Dietary behavior shows no major abnormalities compared to the general German population, except for the energy drinks. The high consumption poses a health risk due to the high sugar content, among other things. Overall, the participants are mostly satisfied with their body image.2 [As shown in previous studies, gamers in Germany are predominantly male, young, and well-educated , 22. The2 . In this2 . As in p2 . Therefo2 . Moreove2 . For addEsports players reported a quiet, positive general health status. But there are significant differences between the desired and actual body image. However, on average this only differs in a maximum of one point, so for the most part they seem to be satisfied with their body image. Being satisfied with the own body is very important for well-being and overall health . RegardiWith 1.6 L per day, water is the main source of fluid for the participants. This is above the general recommendations of the German Nutrition Society (DGE), which recommends 1.5 L per day . The majAlthough it is recommended that water should be used to meet the liquid requirement, other beverages can also be consumed. Very popular in this context are soft drinks. Compared to the 14–17 year olds, who drink an average of almost half a liter of soft drinks per day, the participants in this study drink clearly less . If addiSimilar reasoning applies to energy drinks. On average, participants drink more than one can per week. Compared to the German population, in which only just under 10% drink a can at least once a week, the players in this study drink much more energy drinks . In compBecause the number of studies that investigated the consumption of energy drinks in traditional sports is limited, it is difficult to draw a comparison with them. Nonetheless, one study  looked aIn contrast, the consumption of alcoholic beverages among video game players and esports players is low. About a quarter of the participants do not drink alcohol at all. Compared to the per capita consumption per year of the German population, gamers drink less. Beer alone is already drunk by the German population about 0.2 l per day . In contIn terms of diet, gamers are very similar to the German population, with one-tenth following a vegetarian or vegan diet . In contDairy products are necessary to ensure adequate calcium intake. Gamers, on the other hand, consume less than the recommended amount of 200–250 g milk and two slices of cheese. Overall, the diet is largely carbohydrate-heavy. Several slices of bread, rice, pasta and potatoes are consumed almost daily. Due to the positive effects on the gut, it is recommended to consume whole grain products made from it . In addiEven if the consumption of sweets and chips is quite low, the participants like to resort to quick meals such as fast food. Almost two meals per week consist of it. Due to the negative association with subjective health in this study, fast food consumption should be limited. Even though it is best to consume less, the amount compared to 12–17 year old Germans is similar. They consume about 400–600 g of fast food per week . These rThis study also has some limitations. First, because the current study is based solely on self-reported data, the results may be influenced by social desirability or subjective over- and underestimation of certain behaviors such as sedentary times, physical activity, or food and drink consumption  EspecialSecondly, the COVID-19 pandemic and the accompanying security measures had a significant impact on the population and therefore possibly also on dietary behavior. On the one hand, restaurants were closed during the lockdown, which meant that people were increasingly forced to cook for themselves. In addition, sports in clubs or studios were limited, which affected the sports opportunities and therefore possibly the physical activity. On the other hand, digital media got a boost, which increased screen time and, in particular, video game playing time.Thirdly, questions about the characteristics of video gamers  have been largely self-generated. Either there are no validated questionnaires on this yet or the processing time should be kept as short as possible to reduce the dropout rate. This could have led to inaccurate data. For example, the question on physical activity is based on a simple indication in minutes and hours. For more valid results in future, questionnaires like the IPAQ short form or EHIS-PAQ should be used. Overestimation, misreporting, or the effect of social desirability cannot be ruled out. Accordingly, the high proportion of physical activity compared to other population groups is questionable. Insofar as length and processing time allow, validated questionnaires should be used. The food frequency questionnaire was also slightly adapted, and some foods were combined into one. While portion sizes were included with all foods, everyone could subjectively perceive them differently. However, the results still provide a good overview of the approximate dietary behavior of video game players and esports players.Overall, video game players and esports players reported good general health and well-being, as well as some healthy habits. The gamers seem to eat very similarly to the German population. Nevertheless, there is a need for overall improvement overall. Energy drinks, in particular, can have a negative impact on health and lead to obesity or diseases such as diabetes in the long run. Among other things, sponsors in the industry are responsible for the high consumption. These sponsorships should be critically scrutinized for the possible health consequences of their products. In addition to limiting energy drinks and fast food, the positive effects of fruits and vegetables on health and performance should be highlighted. To keep the target group healthy, they should receive sufficient support and education in nutrition as early as possible.