Passive content fingerprinting is widely used for video content identification and monitoring. However, many challenges remain unsolved especially for partial-copies detection. The main challenge is to find the right balance between the computational cost of fingerprint extraction and fingerprint dimension, without compromising detection performance against various attacks (robustness). Fast video detection performance is desirable in several modern applications, for instance, in those where video detection involves the use of large video databases or in applications requiring real-time video detection of partial copies, a process whose difficulty increases when videos suffer severe transformations. In this context, conventional fingerprinting methods are not fully suitable to cope with the attacks and transformations mentioned before, either because the robustness of these methods is not enough or because their execution time is very high, where the time bottleneck is commonly found in the fingerprint extraction and matching operations. Motivated by these issues, in this work we propose a content fingerprinting method based on the extraction of a set of independent binary global and local fingerprints. Although these features are robust against common video transformations, their combination is more discriminant against severe video transformations such as signal processing attacks, geometric transformations and temporal and spatial desynchronization. Additionally, we use an efficient multilevel filtering system accelerating the processes of fingerprint extraction and matching. This multilevel filtering system helps to rapidly identify potential similar video copies upon which the fingerprint process is carried out only, thus saving computational time. We tested with datasets of real copied videos, and the results show how our method outperforms state-of-the-art methods regarding detection scores. Furthermore, the granularity of our method makes it suitable for partial-copy detection; that is, by processing only short segments of 1 second length. Nowadays, video-based applications flooded the Internet traffic. According to the Cisco Visual Networking Index , by 2019Although current networking systems make efforts to deal with constraints related to real-time computing intensive applications such as mobile processing, cloud-based, and video decoding –7; in viThe Decrease in quality and loss of information: compression, noise addition, contrast/brightness/gamma change, frame rate/bit rate change, frame drop/addition, etc.Post-production or edition: picture in picture (PiP), flip , addition of patterns/subtitlesCamcordingThe protection of copyrighted material is an important issue to address since piracy is a big problem in video distributions. The total cost of piracy is estimated close to US$20.5 billion each year . AlthougAlthough visual transformations are commonly used, also audio component could be easily and independently transformed; for example with temporal attacks, such as speed change, time/pitch shifting and loss of information.The most popularly used transformations in real cases are the insertion of patterns, scale changes and camcording . CamcordTo monetize illegal video distributions are required monitoring and copy detection systems. Video monitoring and copy detection systems are based mainly on passive content fingerprinting and watermarking , 12. In This paper presents a novel methodology to address the video copy detection challenge. For the latter, we focus on two main issues that current state-of-the-art methods do not address or simply struggle with: partial-video copy detection using short segments, and testing over real video copies rather than simulated transformations. Aiming at dealing with these issues, we propose using a set of visual fingerprints whose combination leverages the performance in the partial-video copy detection task, thus outperforming the results obtained by state-of-the-art methods. Additionally, to decrease the computational cost during online processing, we use a multilevel fingerprint extraction and matching process to gradually filter the most similar copies, which saves video frame processing. Moreover, our proposed method uses low memory footprint fingerprints, hence leading to a more efficient searching stage, suitable for large video databases.The rest of this document is organized as follows: in section related work, a brief review from the state-of-the-art about fingerprinting methods is presented. In section proposed method are explained the core processes of our proposal that are video processing, keyframe generation, fingerprints extraction and multilevel matching processes. In section experimental results and discussion, the experimental setup, results, and their respective analysis are described. Finally, concluding remarks are drawn.a.k.a keyframe) is commonly used. Additionally, the frames (or keyframes) may be downsized and color space changed. There are many techniques for obtaining keyframes in the literature [e.g. 1 fps) [In general, the extraction process of a passive fingerprint can be summarized in the following main steps: 1) downsampling the video signal, 2) feature extraction/description and 3) fingerprint compaction if necessary. The goal of the first step is to reduce the processing information decreasing the computational cost of the method. In video downsampling, the selection or generation of a frame that is representative of a video segment (terature –18. Howe. 1 fps) , 20.After video processing, the following steps are for the video fingerprint extraction, which can be generated from global, local, spatial or temporal features. Global features tend to be faster to compute and in a more compact way than local features, desirable characteristics for applications that require real-time processing. In counterpart, local features tend to be more robust against video transformations, whereas temporal features give extra information to persist against temporal desynchronization attacks.e.g. in post-production).For example, in , the colThe use of a Discrete Wavelet Transform (DWT) to generate a global fingerprint is proposed in . That fiIn  two binaOther method which combines global features is presented in . That meLike the global features, local keypoint detectors/descriptors are widely used due to their robustness and discriminability. However, the necessity of descriptor compaction induces the discriminability loss. In general, with a selected combination of feature extraction, fingerprinting methods tend to be more robust against video attacks while increasing the computational cost. For instance, in  four detMoreover, in TREC Video Retrieval (TRECVID) workshop series, a content-based copy detection (CCD) task ran over a period of 4 years (2008 to 2011). The report of those competitions concludes that current state-of-the-art methods, which are robustness focused, are not suitable for real-time applications due to their high computational cost . They alFingerprinting methods are composed of two stages: the offline and the online processing. In the former, the fingerprints of reference videos are extracted and saved in a database. In the latter, a query video segment is matched or rejected. That is, the fingerprints of the query segment are extracted and searched in reference fingerprints database. As it is shown in n frames per second (nfps), and then, the downsampled frames are resized to a fixed length of M × N pixels, where N = 2 × M. To cope with color format changes while we take advantage of the color correlation, we use two models, a transformation to RGB to extract its color correlation histogram as a fingerprint and the intensity of pixels  to extract the other two fingerprints. After video processing, a keyframe is generated per each video segment of length ls seconds. The pixel values from all the frames in that segment are averaged into a single keyframe. That keyframe is first generated in RGB color model KfRGB and then its grayscale version is generated Kftemp.Video processing has two aims: 1) to decrease the computational load and 2) to diminish the effects of video format changes of the transformed copies. With these aims, a video segment is downsampled to Kftemp within its left half, generating a keyframe Kf of size M × M.A very common transformation in copy detection systems is the flip attack. Flip attack changes the position of pixels although not their values. Such localization loss makes most of the descriptors not robust against this transformation. In counterpart, the frame color relation does not change in a flip attack. Due to these facts, we use a frame folding simulation as a counter attack before fingerprints extraction in the gray scale keyframe version, similar to the presented in . The folθstd, then those keyframes are different enough and their fingerprints and timestamps are saved. Otherwise, the strongest set of fingerprints is saved and the timestamp of those segments is updated. Due to local descriptors could be more discriminant under some transformations, the method keeps the fingerprints with a greater number of local features.An important strategy to generate representative keyframes is to detect and merge similar sequential keyframes. To this fashion, after keyframe generation, we extract in advance its fingerprints and we compare them with the obtained fingerprints from the previous keyframe. In our proposed method we use a set of a local and two global fingerprints, whose extraction will be explained in next subsection. If the similarity between each pair of consecutive set of fingerprints is under an established threshold In large video databases, compact fingerprints are desired to optimize searching. As mentioned before, the robustness is also required to avoid false alarm cases. However, there is a tradeoff between robustness and fingerprints dimension. Additionally, in large databases as well as real-time applications, the low computational cost for fingerprints extraction is a concern. Taking into account that existing video transformations and their intensities are very diverse, all these desirable qualities are difficult to achieve. We propose to extract a set of independent fingerprints, each one selected for a specific purpose satisfying the desirable qualities exposed before. We use three fingerprints to describe a video segment: 1) a thumbnail, 2) the color correlation histogram and 3) ORB descriptors. The former two are global fingerprints of a generated keyframe while the last one refers to local features. The set of these fingerprints can be compared easily, due to the thumbnail is a small image, and the other two are binary fingerprints.This representation is taken as a global fingerprint of the keyframe. The keyframe is downsized to (30 × 30) pixels; this small size is sufficient to represent the keyframe without fine details . This isRxy > Gxy > BxyRxy > Bxy > GxyGxy > Rxy > BxyGxy > Bxy > RxyBxy > Rxy > GxyBxy > Gxy > RxyColor Correlation (CC) is defined as the arrangement of red, green, and blue color components in order of intensity . To extrRxy, Gxy and Bxy represent the intensities of red, green and blue components of every pixel  in the keyframe. The resulting six real numbers are truncated, and the first five numbers are stored in a binary form as the feature for the keyframe. The resultant CC fingerprint for each keyframe is represented by only 35 bits. This color correlation histogram is a global feature without any spatial information about the keyframe, and it is robust mainly against scaling, rotation and flipping.Where, It is a binary detector-descriptor proposed in . It has vi of the reference dataset RV is processed and segmented in keyframes. For each keyframe rKfij, where i is the reference video and j the corresponding segment number, a set of three fingerprints rFPij is extracted and stored: 1) rTh which refers to the global fingerprint based on thumbnail, 2) rCC which refers to the global fingerprint based on color correlation and 3) rORB which refers to the local fingerprint (ORB descriptors). Additionally, the timestamp related to keyframe segmentation is saved as the initial and final time for the segment. During online processing, for each query video qvk, the keyframe segmentation is performed with the same process than in offline processing. However, the set of fingerprints per keyframe is extracted by levels according to the filtering scheme presented in Algorithm 1.In the offline processing, each video Algorithm 1 Multilevel matching processqvk find match in rFPij Per each qThLevel 1: extract 1: C = correlation2: ifC > = θThthen3: qCCLevel 2: extract 4:  if There is qCC and rCCthen5:  D = 1 − distance6:   ifD > = thetaCCthen7:   qORBLevel 3: extract 8:    if There is qORB and rORBthen9:    E = match10:     ifE > = thetaORBthen11:     ID, timestamp of rFPij12:      save SimML = wTh*C + wCC*D + wORB*E13:      end if14:     else15:    ID, timestamp of rFPij16:     save SimML = wTh*C + wCC*D17:     end if18:    end if19:   else20:  if There is qORB and rORBthen21:   E = match22:    ifE > = thetaORBthen23:    ID, timestamp of rFPij24:     save SimML = wTh*C + wORB*E25:     end if26:    end if27:   end if28:  end if29: SimML, ID, Timestamp for qvk Return highest correlation refers to the Normalized Crossed Correlation (NCC) between the query and reference thumbnail pair. The NCC coefficient is given by qTh and rTh respectively.In Algorithm 1, distance = mean). The same distance metric is used for a pair of ORB fingerprints (qORBrORB). The value E in algorithm 1 refers to the percentage of total ORB features matched per keyframe, being 1 a perfect score for identical keyframes. To increment the value of E, we define the distance E = 1 if 70% to 100% of the keypoints matched, E = 0.9 if 40% to 69% of the keypoints matched and E = 0.8 and E = 0.7 for the 20%-39% and 1%-19% of keypoints matched, respectively; if there were not matched keypoints then E = 0.To compare a pair of CC binary fingerprints we use the Hamming distance, that is SimML corresponds to the weighted average of individual distances. The assigned weights are wTh, wCC, and wORB respectively.Due to during matching process each pair of fingerprints is evaluated separately, the total similarity score per query In the following subsections, the experimental setup is defined, the video datasets and video transformations used in the experimental process are presented, and a discussion about the obtained results is given.R2015b. Additionally, we employed MEXOpenCV [Experiments were carried out in a PC with Intel i7 processor at 3.4GHz and 16GB in RAM using MATLAB EXOpenCV  which linfps = 5 fps with a short segment of ls = 1 second length. In many related works, it is common to downsize each frame to a QCIF standard size (e.g. [e.g.M × N = 320 × 640 pixels) it is possible to extract more local features achieving better detection scores. In the matching process the corresponding thresholds were: θstd = 0.65, θTh = 0.50, θCC = 0.75, θORB = 0.70. In the first level step a minor threshold is enough to select similar reference videos, whereas the subsequent filtering steps are more judicious. We use adaptive similarity weights per level in the matching process. Those weights are presented in SimML ≥ θML and θML = 0.65.The parameters in video processing were empirically selected. For video downsampling, different rates were used , and, then we generated a keyframe from different video segment lengths . These segments dimensions are based on a standard client buffer length for streaming applications, which is about 6 to 10 seconds . Additioze e.g. , 47). Ho. Honfps For our evaluation we used two public video datasets: VCDB  and ReTRThe VCDB dataset was specially intended to test partial copy detection in real videos, whose attacks were induced by external users to those who collected the dataset, which offers the opportunity to assess fingerprinting methods against non-controlled attacks.Inspired by the experimental setup described in  and aimiAs a reference, the transmission attacks and parameters elaborated in the ReTRiEVED dataset are presented in It is important to highlight that the percentages of found attacks in real copies are different from those in simulated datasets. According to , inserti1-score is the balance harmonic mean between precision  and recall (TPR:True Positive Rate). These parameters are given by TP, TN, FP, FN are True Positive, True Negative, False Positive and False Negative values respectively, and P = TP + TN. Results with F-score tending to 1 are preferable due to effective copy detection methods should avoid false alarm errors.F-score is widely used to evaluate copy detection tasks. FThis dataset contains 184 test videos obtained from eight source videos of different content, characterized by spatial and temporal information and different motions. These videos simulate possible artifacts during transmission; including jitter, delay, Packet Loss Rate (PLR) and throughput. These transmission attacks have a considerable effect on perceived quality , which mCC . As mentST&V Hash . In thisk × k region-wise count of SURF interest points of consecutive frames. This compact form of SURF is designed to efficiently characterize the spatiotemporal content of frames while mitigating the issue of the high dimensionality of SURF features and the excessive computational load to compare them. CST-SURF is promising robust against camcording, edition, and signal processing attacks.CST-SURF . CST-SURIn order to evaluate the detection scores of our proposed method, we compare the robustness of the extracted fingerprints (Th + CC + ORB) against other similar fingerprints: CST-SURF which is based on a compact representation of local keypoints, ST&V that combines visual and temporal compact features, and CC to test the robustness of this employed fingerprint against real or non-simulated attacks. The selected fingerprints from the state-of-the-art are briefly described in the following list.1-scores of the tested different methods under simulated attacks are presented in Tables The obtained F1-scores for the detection of short video segments (1 second length). Particularly, when videos are subject to typical transmission errors such as delay, PLR, jitter and throughput . The results show better detection scores when we use the Th + CC + ORB fingerprints combination than using a unique fingerprint (CC) or than other visual fingerprint combination (SV&V).Transmission artifacts simulated in ReTRiEVED dataset have a considerable effect on perceived quality, due to this fact the detection of video copies based on content fingerprints is easily affected. However, due to the combination of different fingerprints in our proposed method, it can achieve high FOn the other hand, Th prevails especially under decrease in quality attacks. However, this kind of fingerprint has the disadvantage that by itself, is not enough discriminative in near-duplicate videos. This is mainly because this fingerprint does not have the finer details from each keyframe. For this reason, we use this fingerprint as an initial filter to single out similar videos in the multilevel system. On the other hand, the main advantage of the CC fingerprint is that color correlation histograms prevail against many attacks whereas the computational cost to compare this kind of fingerprint in the matching process is very low (the Hamming distance of 35-bit strings). Due to this reason, CC is used as a second filter in the multilevel system. In counterpart, the ORB fingerprint has better results under post-production transformations due to it is based on local keypoints. Hence, the combination of these fingerprints leverages the individual performance of the fingerprint methods. It is important to highlight that our proposed method has satisfactory detection scores even without the use of additional transformation detection and counterattack processes. However, as it was expected, under extreme decrease in quality transformations such as camcording, the proposed method presented the lower detection scores.Some remarks regarding the fingerprints combination of our proposed method are that the global fingerprint TPR = 0.9294 and precision or PPV = 0.5052, which greatly outperforms those results reported in [We additionally used VCDB dataset  as it oforted in  that aresec, extraction of Th = 0.0009, extraction of CC = 0.0017 and extraction of ORB = 0.0034. On average it takes only 0.0440 seconds to process a video segment of 1 second length. Although color correlation is the fastest method, Th + DCT + ORB fingerprints (our proposed method) are more robust against the reported attacks.In Th: is represented by an small image of size 30 × 30 pixels per keyframe.CC: is formed by 35 bits per keyframe.ORB: this binary detector generates features of 32 bytes each one.The dimension of the set of extracted fingerprints is as follows:In summary, the dimensionality of our proposed method depends on the video content information, the changes in video scenes (extracted keyframes) and duration. Th and CC fingerprints have a fixed length per keyframe while ORB fingerprint generates 150 features per keyframe of 320 × 320 pixels on average. Nevertheless, binary fingerprints are faster for searching and matching purposes.We have presented an efficient content fingerprinting method based on the extraction of a set of independent binary global and local fingerprints. The combination of robust fingerprints gives to our method the capability to detect videos severely transformed. Our method aims at achieving robustness against the decrease in quality, edition, and temporal desynchronization attacks which are common in video transmissions. Furthermore, we have presented an evaluation of our method by testing it with real video copies rather than simulations using synthetic and controlled attacks, achieving better precision and recall results than the related works. Moreover, our method can detect short video segments of 1-second length, which is suitable for partial-copy detection.We have also explored the use of light-weight global and local binary descriptors to achieve computational efficiency. In this sense, we propose to combine a global binary fingerprint based on color correlation with a local fingerprint based on the ORB binary descriptor, which brings the benefit of having a low computational cost in the matching stage.On top of this, we propose to use a multilevel matching system that aims at identifying potential similar video copies upon which the binary fingerprints will be extracted, thus avoiding the processing of all the video queries. This multilevel system is based on a filtering stage (using a fingerprint based on the keyframe thumbnail) and a refinement stage (binary fingerprints). According to the performed experiments, using this multilevel system the operations  are reduced to one-fourth in comparison to extracting the full fingerprints set in a brute force search. Furthermore, the searching process can be improved using indexing and/or classification techniques.Our experimental results show that, our method obtains better detection scores than the state-of-the-art methods tested in this work, for all simulated attacks excepting projective transformations, with lower computational time, and processing only short video query segments with a length of 1-second, which is suitable for partial video copy detection. Moreover, this short segment is processed at a speed of 0.0440 seconds in average, independently from the query video frame rate. Regarding the experiments with non-simulated attacks, our method obtains better precision and recall scores than those obtained with other methods in the literature.We have observed that in the case of video copy detection under non-simulated attacks, the local fingerprints lose effectiveness due to the aggressive global transformations suffered by the video frames. This is expected since there is a limitation of robustness for the ORB descriptor against changes in scale, image distortion and illumination changes. In order to address this, the use of different binary fingerprints more suitable for these kinds of attacks could be investigated.


Super-resolution for satellite video attaches much significance to earth observation accuracy, and the special imaging and transmission conditions on the video satellite pose great challenges to this task. The existing deep convolutional neural-network-based methods require pre-processing or post-processing to be adapted to a high-resolution size or pixel format, leading to reduced performance and extra complexity. To this end, this paper proposes a five-layer end-to-end network structure without any pre-processing and post-processing, but imposes a reshape or deconvolution layer at the end of the network to retain the distribution of ground objects within the image. Meanwhile, we formulate a joint loss function by combining the output and high-dimensional features of a non-linear mapping network to precisely learn the desirable mapping relationship between low-resolution images and their high-resolution counterparts. Also, we use satellite video data itself as a training set, which favors consistency between training and testing images and promotes the method’s practicality. Experimental results on “Jilin-1” satellite video imagery show that this method demonstrates a superior performance in terms of both visual effects and measure metrics over competing methods. A video satellite is a new type of ground observation satellite that has been developed in recent years. Compared with the traditional earth observation satellites, it can “stare” at specific targets for a long time to capture continuous “video” rather than a still “image” of the earth scene, which is thus very suitable for dynamic target monitoring. Super-resolution (SR) technology aims to enhance the resolution of images or videos by reconstructing them with higher pixel densities and richer detail information. It is of great interest in computer-vision-related applications, such as biomedical diagnosis, video surveillance, and remote sensing. Because video satellite imagery takes a higher temporal resolution but a lower spatial resolution than a normal remote-sensing satellite, SR finds direct application in raising its spatial resolution.Traditional super-resolution algorithms can be roughly categorized into MISR (multi-images super-resolution) and SISR (single-image super-resolution), in which SISR can also be classified into diversified methods, such as interpolation, reconstruction, and machine learning. Since super-resolution is a typically ill-posed problem, learning mapping relationships between low-resolution (LR) and high-resolution (HR) images is more promising than ordinary interpolation. In the past several years, SR studies have primarily concentrated on machine-learning-based methods, i.e., example-based learning. These methods build a mapping relation between LR images and the corresponding HR images by learning from a training set and then map LR images in a testing set to HR ones. In 2000, Freeman et al.  proposedWith the popularity of deep learning, super-resolution has recently passed a fast development period. In 2014, Dong et al.  pioneerei et al.  proposedi et al. , and thei et al.  fused sei et al. , thus rei et al.  combinedi et al.  designedi et al.  introducIn the era of big remote sensing data, the question of how to efficiently transfer, process, and store massive remote sensing data has become a common concern for researchers . The supVideo satellites have greatly improved the dynamic monitoring capability of satellite remote sensing systems in some typical scenarios, such as resource census, disaster monitoring, ocean surveillance, dynamic target tracking, and dynamic event observation. However, various constraints on the process of satellite imaging happen under a video satellite environment, including but not limited to low spatial resolution due to ultra-long distance imaging, sensor noise, atmospheric disturbance, and image degradation caused by relative motion. Especially, the optical imaging system has to reduce its spatial resolution in the exchange of continuous video, and meanwhile the communication system has to impose heavy data compression to guarantee reliable information transmission under limited sky–earth channel capacity. These factors result in low video quality in terms of spatial resolution and clearness. Therefore, video satellite applications strongly call for techniques, such as super-resolution, to enhance spatial resolution while preserving high temporal resolution.However, super-resolution for video satellite imagery is confronted with huge challenges due to the inherent characteristics of satellite video. In contrast to normal still satellite imagery (at the sub-meter level), the dynamic imagery captured by a video satellite has inadequate ground spatial resolution (at the meter level). A pixel within a video satellite image may contain a variety of features and the homogeneity of the image block is much weaker. Moreover, since satellite video covers varieties of objects with different spatial scales, homogeneous landform regions with similar textures and appearance structures show statistical diversity in spatial distribution. Therefore, the traditional methods based on a dictionary or example learning are less-well adapted to this diversified image content and activity complexity for precise super-resolution reconstruction.The recently developed deep convolutional-network-based methods, typically SRCNN  and ESPC(1)We propose a five-layer end-to-end network structure without any pre-processing and post-processing for the sake of simplicity. As opposed to outputting a high-dimensional feature layer directly and post-processing as in ESPCN, we place a reshape or deconvolution layer at the end of the network to retain the distribution of ground objects within the image.(2)We employ a different strategy for the loss function: unlike other CNN algorithms that simply calculate loss via output and ground truth images, we create a joint loss by combining output and high-dimensional features of a non-linear mapping network. This operation can take into account layers before and after magnification, which facilitates a more precise mapping relationship between LR and HR images.(3)In training, we use satellite video data themselves rather than other images to construct training set. This strategy contributes to the consistency between training and testing images in terms of image content statistics, thus enabling the practicality of the algorithm.The main contribution of our work can be summarized as follows:In 2015, the Changchun Institute of Optics, Fine Mechanics, and Physics successfully launched the “Jilin-1” video satellite of 1.12 m resolution, which is really a breakthrough for Chinese ground observation satellite systems. In this article, we use these video data for super-resolution by means of the proposed convolutional network. For better reconstruction performance in super-resolution applications, we choose video satellite data itself as the training set instead of other image sets. Video frames are more beneficial to the reconstruction of the video itself due to the consistency between training and testing frames in statistical distribution. Thus, our method, by making full use of the video data itself, does not require an extra training image dataset (note that training data and testing data are excluded). In fact, because of the very rich types of ground objects in remote sensing images and different features exhibited at different resolutions, no uniform training dataset for the super-resolution of remote sensing images has been created yet. The use of video satellite data for super-resolution reconstruction can just avoid this problem.The procedure of our method consists of the following steps. First, we randomly extract a fraction of the video frames from “Jilin-1” video as a training set. Then, we choose images from the rest of the video frames as testing ground truth images  has a higher demand for spatial coordinates and radiation compared with natural images; in other words, the reconstructed remote sensing images should be “more clear” in content and be “more accurate” in coordinates for ground objects. Therefore, the entire non-linear mapping net is designed in a shrinkage-expansion manner, where the size of the filter is firstly reduced to pixel level and then grows larger. After reconstructing feature layers, the network achieves pixel-wise prediction for LR image input.Reconstruction and Output: This part transfers the feature layer of the non-linear mapping network into an HR image. For comparison, we design three ways for reconstruction and output in our algorithm. At first, learning from ESPCN, we output a high-dimensional feature layer directly (marked as We take the Mean Squared Error (MSE) to calculate the loss function, which is widely used in the super-resolution problem. There are differences for the three kinds of output. For the first way For the sake of clear illustration, the used pixel-rearrangement operation is shown in For the last two ways, we first calculate the output with the ground truth image as follows:Further, as shown in We choose two commonly used evaluation metrics, PSNR  and SSIM  to measure the quality of the reconstructed images. They both evaluate differences between a reconstructed image and a ground truth image, but make differences in terms of visual perception.PSNR, one of the most widely used evaluation indexes in computer vision, is calculated based on the error between the corresponding pixels. It can effectively measure error at the pixel level. However, it may appear that the evaluation results are inconsistent with human subjective feeling as it does not take the visual characteristics of the human eye  into account.Instead, SSIM measures three aspects of image similarity, including brightness, contrast, and structure. Unlike PSNR, it is based on structural similarity rather than error sensitivity. The larger these two indexes, the better reconstruction we have.Both the training and the testing data used in this article were based on “Jilin-1” video satellite imagery. We extracted video frames and selected one for every five frames  as training data, and we selected test images from the rest at random. We chose several areas in different countries with certain typical types of surface coverage, including vegetation, a water body, and a variety of buildings. The convolution sizes of the whole network were respectively set to 5, 5, 1, and 3 with the numbers of filters being 64, 64, 32, and 27. Note that ReLU is appended to each convolutional layer. We compared several common optimizers  for the purpose of faster convergence in training and better performance, and finally decided to take AdamOptimizer  with weiWe use bicubic interpolation as our baseline and compare our method with SCSR  and SRCNWe notice that the original code published by Dong (running in Caffe) cannot obtain satisfactory results when using video data as training data. As we can see in From the perspective of quantitative evaluation indexes, our network produces better results compared with competing methods, which are shown as tabulated results in As shown in In this work, we have proposed a super-resolution method using a convolutional network for “Jilin-1” satellite video data. We designed a five-layer convolutional network with three kinds of output, which is trained by a novel joint loss function. We also use satellite video itself as training set to replace other remote sensing training sets, which enhances the practicality of the method. Experimental results on real-world satellite video data show that our method yields a boosted performance in both objective metrics and visual quality.


The lack of a theoretical foundation and definition of the notion has not hindered actual research. My aim here is to identify how active scientists use the notion of neural representation, and eventually to list a set of criteria, based on actual use, that can help in distinguishing between genuine or non-genuine neural-representation candidates. In order to attain this objective, I present first the results of a survey of authors within two domains, place-cell and multivariate pattern analysis (MVPA) research. Based on the authors’ replies, and on a review of neuroscientific research, I outline a set of common properties that an account of neural representation seems to require. I then apply these properties to assess the use of the notion in two domains of the survey, place-cell and MVPA studies. I conclude by exploring a shift in the notion of representation suggested by recent literature.The word representation , and many of its related terms, such as to represent, representational and the like, play a central explanatory role in neuroscience literature. For instance, in “place cell” literature, place cells are extensively associated with their role in “the representation of space.” In spite of its extended use, we still lack a clear, universal and widely accepted view on what it means for a nervous system  The word “representation” , and many of its related terms, such as “to represent,” “representational” and the like, play a central explanatory role in neuroscience literature. For instance, in “place cell” literature, place cells are extensively associated with their role in “the representation of space.” Lately, the notion of representation has gained more visibility since the development of new methods that try to translate brain mapping markers  into stimuli features, and vice versa. Indeed, the development of methods such as multivariate pattern analysis (MVPA), which attempts to identify the contents of brain activity, has put the notion of representation in the center of the debate.on representation, no scientist knows how representations represent”  that has allowed authors to use the term without worrying much about details.In this sense, whether or not it is useful to analyze the notion of “neural representation” is a legitimate question, as researchers use the term without having a working model of what it corresponds to and yet still produce relevant scientific work. However, in my opinion, the project is both legitimate and useful. The notion of “neural representation” is central to neuroscience, because, in the majority of studies, it corresponds to “what is to be explained,” such as, characterizing what a spike train corresponds to, and or “how we explain it,” by using representations to explain, for instance, how animals navigate efficiently. In short, the notion of neural representation plays a central role in explanations of the neural mechanisms underlying brain function. This means, among other things, that assuming a particular notion of representation constrains the questions one asks experimentally, the design of the experiments themselves and the interpretation of the results. If scientists can do without a clear idea about the notion, it is not because it is absent from their work, but because they implicitly assume a particular model of the notion. Hence, in my opinion, exploring such implicit models of the notion, and what they imply, is necessary and timely. To consider that the brain can code for objective features of the environment such as an Euclidean coordinate map of geographical space, or to consider that the brain cannot code for objective features of the environment point to quite divergent models of brain function. The experimental questions, designs and interpretations of assuming one or the other approach develop completely different frameworks. Therefore, in my opinion, it is critical to understand the model of “neural representation” that neuroscientists assume, and to characterize what the model implies empirically.Such is my aim here: to carry out a model-dependent analysis of “neural representation,” that is, an analysis whereby the experimental findings in neural-representation studies are supposed to fit the theoretical model derived from the authors’ replies, and the literature on the topic. Specifically, I first show the results of a survey of authors of empirical studies in two domains—place-cell and MVPA research. This allows me to identify what such authors have in mind when they use the notion. Based on their replies and on a review of their research, I outline a set of common properties that an account of neural representation seems to require. I then apply these properties on the two domains of the survey: place-cell and MVPA studies. I conclude with a speculative note based on recent theoretical approaches.I carried out an online survey of authors of studies in two domains: place-cell research and MVPA research. These two domains examine neural activity at very different levels of analysis, using very different methodologies and techniques, but sharing the notion of “neural representation” as a central element of their research. On one hand, under the label of “place-cell research,” I classify studies that focus on the firing characteristics of hippocampal neurons in awake and free-moving animals. These sorts of studies has provided data associating environmental stimuli and behavior with neuronal activity. The electrophysiological data capture a number of different dynamics of activity and coherence among groups of cells, including “place cells,” which have a preferential spatial location and “grid cells,” which produce a preferential periodic array of locations’ activity, as well as other neurons associated with spatial activity, such as “head direction” or “border” neurons. On the other hand, under the label “MVPA research,” I include studies that use multivariate statistical tools, known as classifiers, to characterize the encoding and decoding of environmental features in BOLD-based activation patterns. Both field studies share the aim of understanding how a brain makes sense of the world based on a comprehensive model that encompasses cellular, neural systems and behavioral data using different methodologies and units of analysis.I identified candidate studies by searching for peer-reviewed publications published between 2010 and 2015 in Scopus. For MVPA research, I used the following search string:∗”) AND (TITLE-ABS-KEY (“MVPA”) OR TITLE-ABS-KEY (“multivoxel”) OR TITLE-ABS-KEY (“multi-voxel”))PUBYEAR > 2009 AND PUBYEAR < 2016 AND TITLE-ABS-KEY (“fMRI”) AND TITLE-ABS-KEY PUBYEAR > 2009 AND PUBYEAR < 2016 AND TITLE-ABS-KEY (“place cells”) AND TITLE-ABS-KEY ∗In your opinion, can representation be measured?∗If so, what would be the measure of representation ?∗What unit of analysis should be used ?The answers of all the authors will be treated anonymously. At most, if the survey is eventually published, I might include, for illustrative purposes, some literal definitions, without indicating its author.I introduced the preference of intuitive definitions, because the survey was intended to reveal the actual uses of the notion and to explicitly avoid the temptation of relying on dictionary definitions. Additionally, I decided to include some examples, such as “mutual information” or “local population of cells,” because in a pilot survey that I had carried out among colleagues of mine who did not participate in the survey, I found that readers had some doubts about what I was referring to as “measures of representation” or “levels of analysis.”The emails were sent from March 28, 2016, until April 8, 2016. There were 102 responses (38.4%) that included replies to the questions and 14 refusals (5.2%), mainly attributed to a lack of time. There were 150 (56.3%) emails that went unanswered.This study was carried out in accordance with the recommendations of the Comissió d’Ètica en l’Experimentació Animal i Humana (CEEAH) at the Universitat Autònoma de Barcelona. Consent was obtained by email from the survey participants.First, the authors’ replies reflected a general concurrence that a general theory of neural representation was lacking. As one author put it bluntly: “We don’t know how the brain represents.” However, there are some basic aspects of the notion about which there seemed to be some sort of agreement.Regarding the question of definition, there likely were as many definitions as there were authors’ replies; however, all but one of the authors endorsed the idea that neural representation demands two components: a neural component and an environmental component. The neural component is usually referred as a “pattern of neural activity” or simply “neural activity,” as well as related terms such as “neural substrate,” “neural response,” or “spike activity.” Only fifteen authors did not use a neural term to refer to the internal component. I will return to this point below, upon describing the different conditions for neural representation. For the environmental component, there was a wider variety of terms, among which we could determine three main semantic domains. First, there were those authors who used expressions such as “external world,” “outside world,” and “the environment;” second, there were those authors who used terms such as “stimulus,” “stimulus features” or “stimulus properties;” and, finally, there were those authors who used terms such as “behavior,” “state of the animal” or “dimension of experience,” although in this case, the qualification of “environmental component” is less clear, since it could include internal elements such as, “hunger” or “pain,” for example.stands for the environmental one. The exact wordings of the replies could be further classified into four different, but probably not incompatible, approaches. First, there are authors who used verbs such as “point” or “designate.” These authors seemed to embrace a neutral position in which the representational relation would only be defined by the connection between the neural component and the representee, leaving open the specifics of the relation. Second, there were the authors who defended the idea that to represent is for a neural activity to carry information about the environmental feature that the neural activity is supposed to represent. This description corresponds to the model suggested in the few extant theoretical models of neural representation: “a representation can be understood as a signal that is used in a transformation of information”  the pattern of neural activity; (b) the environmental representee; and (c) the association between the two. I will address each one in turn, and I will then use them to assess the two domains of focus in this paper: MVPA and place-cell research.As I indicated above, the most common idea among authors was that neural representation corresponds to some pattern of neural activity. Nevertheless, the authors seemed very flexible and pragmatic in accepting what type of neural activity might underlie a certain neural representation. For the unit of analysis, only one author considered that the notion of representation belongs exclusively to the conscious level, whereas the rest adopted a pragmatic and comprehensive approach addressing how the attribution of representational properties is a contextual criteria that depends on the feature or stimulus to be represented: different features are represented in different levels of analysis, from cells to networks. Among the different replies, such expressions as the following were common: “any level of analysis can work ,” “all analysis levels of the organization of the nervous system can be considered pertinent,” “the unit of analysis is not fixed, functional roles can be attributed at any descriptive level,” “representations can occur in single cells, small networks, and across the entire brain,” and “the unit of analysis can again be anything, as long as is clear that any claims are also made on the level of analysis.”In contrast, with the flexibility found relative to the type of neural activity considered to reflect neural representation, the authors and extant literature have been quite strict in the operational and formal features that any candidate should exhibit. I have summarized the features that a pattern of neural activity must exhibit to be endorsed as a neural representation in the following three conditions:(i)well-formed.A pattern of neural activity must be (ii)self-contained.A pattern of neural activity must be (iii)neurally sound.A pattern of neural activity must be well-formed.” I have borrowed the notion of “well-formedness” from the domain of linguistics. In linguistics, a well-formed sentence is one that is judged to be correct according to the set of rules and constraints of the grammar under scrutiny. In transferring the term to the domain of neural representation, the idea is that the representational competence should be based on a specific “grammar” that explains how constituent parts must be formed, as well as what rules and constraints should be followed to build each particular pattern of neural activity. More importantly, the grammar also explains how a particular pattern, based on the combination of the basic features of the neural activity, relates to specific features of a particular representee. For instance, in the case of place-cell research, the grammar of the representational roles is based on such features as timing  or spike rate. Note, however, that at present, the grammars suggested for place cells or MVPA patterns are merely hypotheses, because we still do not know the codes that are actually used by the brain. In any event, extant grammars are sufficient to systematically identify a specific pattern of neural activity as a possible neural representation and to distinguish it from other neural activities that are considered—for instance, noise.The first condition required of patterns of neural activity is for them to be “self-containment,” I mean the property of being able to fulfill the representational role regardless of whether the representee is present. The self-containment condition is a fundamental property for neural representation. Classically, it has been assumed in all representational models of cognition  is a correlate of the representational activity. This is an implicitly held assumption that can be derived from the authors’ replies or extant neuroscience. For example, as one author put it: “We obviously need first to understand how our variables correspond to the neural activity that underlies a particular representation.” More specifically, as another author maintained: “What we call neural representation based on our measurement techniques may have little or nothing to do with how relevant information is actually being processed in the brain.” For one thing, the development of new techniques and methods of revealing brain activity results in a number of measures that are correlated to neuronal activity. However, the connection of these correlations to the actual neural activity that underlies the suggested representation is in some cases still a matter of discussion. The problem is that without a sound account, research runs the risk of losing sight of what the variables really measure. Therefore, the need for a justifiable rationale of the neural basis of the approach is especially necessary.Finally, we have the By “representee,” authors tend to refer to some element of the environment of the individual. Common to what all authors included in such a category is some mention of stimuli, in general, or of particular features or properties of relevant stimuli. The fact is that there is no special choice in the category or scale for a representee; it can be an object, the slope of a line, a behavior, a place, etc. The sole inclusion criteria that authors seemed to support for representees are that the candidate must be relevant for the studied organism and that the organism must be in possession of mechanisms that can detect such a representee. Categories and scales notwithstanding, the authors’ replies and extant research have suggested that there are certain requirements that the representee has to satisfy to be accepted as a suitable candidate. I have summarized these requirements in the following properties:(i)measurable magnitudes.Representees must be (ii)self-subsistent.Representees must be (iii)meaningful for the organism in question.Representees must be measurable requirement, which states that the representee must be some type of physical magnitude present in the environment  is there or not. We must be able to say, “there is a rabbit,” “there is no rabbit,” or “there is a fox, as a basis for asserting,” “the organism has correctly represented the rabbit as a rabbit” or “the organism has misrepresented the fox as a rabbit.” Therefore, the representee must be autonomously characterizable; otherwise, the representee cannot be represented at all.The meaningful requirement. The issue of meaningfulness is widely accepted as critical and has been subjected to very extensive analyses. In general, by “meaningful,” authors seem to suggest that the representee has to have a certain biological value  or individual sense for the organism. For instance, some replies concurred with the idea that representees can only correspond to “meaningful dimensions of experience ,” while another maintained that “whether or not (…) a ‘representation’ [in the sense of representee] has any relevance to the operation of that brain requires further empirical enquiry [than simply attesting the association]. Some ‘representations’ will be merely epiphenomenal, indicating something to the experimenter that has no direct relationship to the operation of the brain.” Hence, the need for this condition comes from the fact that the environment offers an open set of ways of being characterized, and only a determinate sub-set of those are the ones that organisms use as sources of knowledge. These elements are therefore the ones that are supposed to be “represented.” The problem is that the meaningfulness of the environment for a particular organism is not a given. If the scientist selects some piece of the environment because she simply believes it is relevant, without independent evidence apart from an educated guess, she runs the risk of establishing a spurious correlation. As I indicated in the case of the neurally sound condition, the versatility of the new techniques and methods of correlation between brain activity and environmental magnitudes makes it relatively easy to find correlations for reasonable ways of segmenting the environment, regardless of its meaningfulness for the organism under study. For example, assume that a scientist believes that “things that weigh less than 125 g” is a possible representee. Moreover, assume that she finds a neural component that reliably and solely correlates with “things that weigh less than 125 g” and that such a neural component complies with the well-formed, neurally sound and self-containment requirements. Then, if we do not introduce the meaningful condition, she will be licensed to consider it to be a good candidate for representee. Of course, it could certainly be that “things that weigh less than 125 g” would be a meaningful representee, but the point here is that the choice of the representee must be based on as much independently based evidence as possible. The choice of a representee should take into account its biological value to the organism for that particular type of situation or function, the sensory and cognitive endowment of the particular organism, and the developmental stage of the organism, as well as other factors. In very simple organisms, meaningfulness can be associated with feeding, protective or reproductive elements. Obviously, as we get to more complex organisms, the notion of meaningfulness becomes more loose and, therefore, more difficult to assess. In general, though, we could say that the piece of the environment that has meaning must be inferred from the way the organism responds to such an element, that is, attesting that it has causal consequences in the behavior of the individual. In contrast, if a particular piece of the environment is singled out but has no impact on the behavior of the individual, then it would be advisable to avoid it.Finally, we have the relation between the patterns of neural activity and specific representees. It is the representational relationship that defines an account of representation. As we have seen, though, we still lack a general theory of neural representation and, logically, of the representational connection. Therefore, it would also seem difficult to identify how specific empirical accounts of representational associations should look. However, as we have seen, neuroscientists seem to overlook this difficulty by assuming that the representational relationship exists and that their task is to identify a possible relation between two components, regardless of the nature of the relation in question.Any account of neural representation is based on the In general, the different representational approaches (MVPA and place cell research) use statistical measures to identify relations between neural activity and environmental magnitudes. Indeed, this type of statistical correlation is the basis for nearly all uses of representation in neuroscience. Furthermore, according to the survey responses, the authors were very flexible with regard to the specific statistical measures used. The survey and extant literature seem to show a pragmatic flexibility regarding the ways of identifying and characterizing the representational association. Generally, any measure that identifies a correlation between a particular representee and a particular pattern of neural activity is acceptable, including mutual information, similarity analysis and the like. Common among the replies were expressions such as the following: “Any measure that determines the degree of mutual information can be used” or “representation can be measured in many ways, including both linear correlation and information theory measures.”reliable. I understand this term “reliable” to refer to the methodological condition of showing that the correlation be consistent and replicable. This requires that the approach prove that, under the same method in different circumstances and under different methods in the same circumstances, the correlations will identify the same patterns of neural activity for the very same representees  tools have been used to map stimuli features into BOLD signals across the brain and vice versa. The ambitious scope of the approach has led authors to use the label of “neural mind reading” in describing such studies. A neural reading MVPA analysis usually involves a three-space model: the stimuli input information, the feature space, and the BOLD signal. A visual stimulus, for instance, is characterized along axes that correspond to the luminance of each pixel, and a natural scene is represented by a single point in the input space. In such a feature space, each axis corresponds to a single feature, and each stimulus is represented by one point in that feature space. Finally, the activity space represents the activation of all the voxels within an ROI (region of interest): the axes correspond to the individual voxels, and ROI’s activation pattern is identified by a unique point in the activity space. In general, the transformation from the input space into the feature space is a non-linear mapping, whereas the transformation from the feature space to the particular BOLD activity, or vice versa, is a linear mapping. In this later case, the mapping involves training a linear classifier that allows mapping multi-voxel activation patterns onto specific stimulus labels.The MVPA approach has already yielded impressive results, but does MVPA withstand the scrutiny of our operational definition?well-formedness, MVPA research seems to satisfy the condition. It is true that, as I said above, the grammar defining the particularities of well-formedness should be derived from the theory of neural representation supported by the specific approach . However, as stated previously, we still lack such a theory. MVPA has assumed the idea that even if we do not know how representations work, we know they exist. Hence, the task is to reveal them, and MVPA does this in a systematic and formal way that can be considered to be a grammar of sorts by prescribing what the basic constituents of a pattern of neural activity should be and how such constituents are combined to build a particular pattern. Specifically, MVPA departs from the BOLD activity of single voxels by creating multivoxel patterns of activity, and these patterns are then treated as points in a multidimensional space with as many dimensions as voxels there are in the analysis. If we have only two voxels, then a particular pattern can be conceived as a line on a plane based on each voxel’s activity. For more than two voxels, the plane reflects higher-dimensional space. Then, each multivoxel BOLD activity can be transformed in a multidimensional pattern, and all such patterns can be classified in, for example, two different conditions. It is true there are different versions of the grammar employed that, among other things, depends on the type of classifiers—that is, the algorithms classifying the data into conditions. In any event, each version provides its own grammar, with specific rules and constraints, therefore making it possible to assess the well-formedness of its neural representations. Therefore, MVPA can be said to comply with the well-formedness condition in what concerns the form of the pattern of neural activity, awaiting an understanding of how such a form is the outcome of the representational encoding.As stated above, the conditions for patterns of neural activity to be considered neural representation are soundness, well-formedness, and self-containment. For self-containment, that is, the condition that assumes that the pattern should comply with its role regardless of the representee being there. Despite the dearth of studies, there are nevertheless a few that have directly probed this condition. Specifically, the authors of these studies have applied MVPA methods to decode imagined movements  contribute to the BOLD signal, and the functional implication of the specific neural activity underlying a BOLD signal is not comprehensively known. Therefore, much more precise information about the relationship between the BOLD signal, neural activity and cognitive processing is needed before we can rely on BOLD-data analysis to understand the neural activity being tapped when using MVPA techniques. Likewise, it is still unclear how changes in BOLD signals across time are to be correlated with neural activity. In this sense, the temporal and spatial resolution of the BOLD signal is not adapted to neuronal activity. In the window of time that a BOLD signal is characterized, many relevant neural activities implicating many different neural representations might have taken place without permitting detection by MVPA classifiers. Furthermore, we still do not know at what level  the information relevant to the task is processed, and it could be the case that the measured signal is irrelevant for probing the informational level , and how such parameters vary in the chosen stimulus ensemble can lead to different results. Therefore, the representee is not a given, but a chosen. Even in the case where the stimuli are transformed into a set of dimensions, such as pixel luminance, the scenes to be classified are chosen by the scientist. This has to be taken into account, because classifiers are so powerful that they are capable of finding correlations between nearly any pattern and any feature. Hence, classifiers run the risk of mapping a set of features that have no representational relevance. Scientists can easily choose an arbitrary set  and find reliable patterns of activation that correlate with any such features. The risk of inferring spurious representees has been discussed by some authors , but there are also others based on nearest neighbor, Bayesian analysis, linear discriminant analysis, and multinomial logistic regression, among others. As I indicated above, the authors seemed to be very flexible in accepting any technique shown to enjoy suitable and robust technical performance. However, as some authors have indicated, it is one thing for a method to technically perform; it is another to assert the scientific relevance of its findings. As one interviewed scientist phrased it, “It is not enough to show that a particular algorithm finds a correlation between a candidate representation and the represented object; we need independent evidence to sanction such a correlation as genuine.” In the case of MVPA, the challenge is to show that the actual algorithm employed is irrelevant with regard to the findings—i.e., that the MVPA approach is, in general, On one hand, “reliable” means that different MVPA algorithms should yield the same results with the same datasets; otherwise, the pattern of neural activity corresponding to a particular neural representation would be algorithm-dependent. In such a case, we would have as many patterns as there were algorithms available, which would undermine the reliability of MVPA as a tool to identify neural representations. On the other hand, “reliable” means that MVPA classifiers should identify the same patterns of neural activity in all the instances of its representee; otherwise, MVPA would be context-dependent, and that would imply that the findings could not be reliably interpreted as a marker of particular representee but instead as a marker of the representee-in-context. Regarding the former aspect of reliability, technique-dependence cannot be characterized as black and white. The choice of a particular algorithm depends on a number of factors, beginning with the question we want to answer , particularities of the task , and the protocol , as well as upon many other factors. Therefore, it might not be useful to compare two classifiers in the same circumstances. However, regardless of this caveat, there are already some studies that have begun assessing how different classifiers identify similar patterns of neural activity for particular representees , with prRegarding the question of context-dependence, some authors have made observations leading them to argue that MVPA findings are indeed context-dependent. However, this is an empirical question that can only be resolved with further studies. Moreover, it is conceivable that careful experimental designs can provide evidence of context-independent MVPA methods, as some have already shown .After decades of research and numerous studies, place-cell research has demonstrated the existence of some type of association between information in the environment and the rate and/or temporal spike activity of neurons. When an animal moves through the environment, certain neurons preferentially respond upon passing through a specific location. In the specific case of place cells, networks of neurons in the hippocampus show preferential activity in particular spatial locations, or place fields. Such information is so precise that scientists can accurately estimate the rat’s position in space by observing the rate coded output of just 50 simultaneously recorded place cells. Numerous studies have confirmed and extended these observations  correspond to the neural correlates of a candidate representee. “Well-formedness,” in turn, refers to the rules and constraints that should underlie the characterization of patterns of neural activity. Finally, the “self-containment” condition refers to the requirement that patterns of neural activity should show the capacity to substitute the representee in the internal workings of the brain, without the representee being present. For representees, the proposal presented here suggests that representational accounts should select only meaningful, self-subsistent and measurable representees from the set of all possible segmentations of the active environment. Finally, the methods for establishing the relation between the representation and the representee should be reliable—that is, correlations should be context- and technique-independent.Table 1 for a summary of the assessment). I have shown that MVPA satisfies, at least partially, the conditions of well-formedness and self-containment for the patterns of neural representation, while still failing to fully meet the challenges of neural soundness. MVPA still needs to prove, in turn, that it is discriminative enough in relation to identifying only meaningful representees, and it also must show that it is context- and technique-independent. We have also seen that place-cell research does partially comply with the neural soundness condition for patterns of neural activity. It must also seek to meet the well-formedness condition by establishing a grammar for the neural code, and it also remains to be seen whether it will satisfy the self-containment condition. Concerning the criteria for representees, place-cell research seems to fully satisfy the measurable, meaningful and self-subsistent conditions. Finally, place-cell methods of establishing the relation between patterns of neural activity and representees fully complies with the reliability criterion. We can conclude that extant research on representations partially satisfy the conditions for neural representations and that they would only need more evidence, better neurophysiological understanding, and specific neurobiological and conceptual constraints.I have used this set of conditions to assess extant MVPA and place-cell research (see A possible objection to the previous analysis may be the degree to which it can be generalized, as I only apply the analysis to two research domains: place-cell and MVPA studies. While I cannot prove my analysis can be generalized to all the domains where neural representation is used, I believe that there are indications that this could be the case. For one thing, I intentionally chose two of the more methodologically and theoretically distant domains in neuroscience, place-cell and MVPA, so as to examine the two more distant uses of the notion of “neural representation.” Place-cell settings function at the level of cell activity, while MVPA focus on patterns of brain activity that encompass the whole brain. Moreover, place-cell research associates the animal’s real behavior with direct electrophysiological data; in contrast, MVPA applies multivariate statistical tools to connect stimuli features with indirect measures of brain activity. However, what place-cell and MVPA research have in common is, namely, processing environmental activity in order to carry out cognitive processes that are connected to behavior, which is something that is shared by many other domains in neuroscience, from animal behavior, up to the more sophisticated cognitive processes of humans. The fact is that the brain does what it does, regardless of the different ways we have of studying it; hence, the explanation of its function cannot be technique-dependent. The notion of neural representation, even if analyzed in specific domains, must possess general and shared properties that concern all levels of analysis. Therefore, I believe the analysis presented here can be generalized in the sense that it can be applied to any domain where the notion of “neural representation” is used.Moreover, the analysis presented here has opened an intriguing possibility that may have interesting consequences for the notion itself. A careful analysis of place-cell multidimensional sensitivity presents a serious challenge not only to the evaluation of place-cell representees but also to the notion of neural representation in general. Let me explain. During decades of research, the simple representational model of place-cell studies has been based on the assumption that the hippocampus was involved in processing a spatial map of the animal’s environment. This hypothesis has been favored because location is the most informative and temporally stable feature in the environment; therefore, it is highly predictable . As someI will now examine how this shift affects the notion of representation. Let us consider the contextual, sequential and predictive functional interpretations of place-cell activity that we have examined above. Context is generally understood, in the area of hippocampal research, as any situation defined by a coherent set of features occurring within a particular environment . On the My view is that any of the previous hypotheses constitutes a direct challenge to the classical notion of representation by making the classical view useless for understanding what place cells actually do. As I mentioned above, a representation involves a coding, mapping or referring relation between some neural activity and some environmental component. Nevertheless, contexts, sequences or predictions do not seem to be descriptions, maps, or copies of the environment as such. Moreover, they are not self-subsistent elements of the environment, because they do not exist independent of the ongoing animal-environment interaction. Location, for example, does not seem to be treated by the hippocampus as a re-presentation of the environment but as a spatial marker related to the contingencies of the situation; it is one of many dimensions in multidimensional processing that involves spatial features, but it also taps into the motivational state of the individual, the features of the task at hand, the history of similar interactions and many other informational elements relevant for the individual in such a situation. Contexts, sequences or predictions are better understood as complex information states that integrate different sources of information, external and internal to the animal, whose function seems to be structuring the interaction of the animal with the environment in that particular situation. Therefore, the model of representation of a neural activity engaged in mapping, copying or referring to some environmental element cannot characterize the contextual, sequential or predictive processing; the hippocampus seems to be doing something not specifically representational with the information.representing the environment to guiding action through anticipation. Instead of mapping or encoding processes, the hippocampus might be set to control the contingencies of a situation by efficiently integrating environmental, cognitive and motivational information into a processing state ready to guide the individual in that particular situation. In other words, the hippocampus could be better seen as a system with the function of providing a dynamic state space where “the set of probable events and contingencies” (gencies” , p. 700 gencies” , p. 1199Accordingly, instead of computing “representational” processes, the hippocampus would be better described as computing “situational” processes. The term “situational” has already been used in the sense of a cognitive construction meaningful to the animal only in the situation in which it is involved and probably related to helping the animal to address it . Hence, It is important to note, however, that the hypothesis suggesting that the hippocampus processes situations instead of elements of the environment is not exclusive. It could very well be that the hippocampus has overlapping modes of processing and that one mode could represent objective coordinates of the spatial layout of the animal. This is obviously an empirical question. Moreover, the situational approach might not have any relevance outside the hippocampus; thus, it might not affect MVPA research. The point is that place-cell research evidence so far might be better understood within the situational approach. In any event, the suggestion of a situational model obviously requires further investigation and development.As presented above, neural representation is a central and non-trivial notion in extant neuroscience. Authors extensively use the notion, putting important explanatory weight on it. However, no agreed benchmark against which to assess specific theoretical and empirical claims exists. Based on the survey and previous literature, I have outlined a set of properties with which an operational account of neural representation should comply. The idea is that one could use such properties as a way of assessing a particular claim of representation. However, the evaluation of place-cell research under such criteria raises the intriguing possibility that the neural activity of place cells might be better understood at present within a non-representational framework. The representational scheme based on the idea that neural activity encodes, maps or refers to self-subsistent, meaningful and measurable features of the environment might not be the best way to construe the role of place cells in hippocampal function. Instead, a situational approach, based on the integration of spatial and non-spatial information with the aim of creating a state space where the contingencies of a situation are processed, seems to be more useful for understanding place-cell activity. The challenge now is to explore and develop this situational approach.OV conceived the article, carried out the survey and its analysis, and wrote all the versions of the manuscript.The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Similar results were observed for malnutrition identified by SGA and NRS-2002. A sensitivity analysis by including patients with acute illness showed that albumin and prealbumin concentrations were dramatically reduced, indicating that they must be carefully interpreted in acute care settings. This review showed that BMI, hemoglobin, and total cholesterol are useful biomarkers of malnutrition in older adults. The reference ranges and cut-offs may need to be updated to avoid underdiagnosis of malnutrition.Malnutrition is a common yet under-recognized problem in hospitalized patients. The aim of this paper was to systematically review and evaluate malnutrition biomarkers among order adults. Eligible studies were identified through Cochrane, PubMed and the ProQuest Dialog. A meta-regression was performed on concentrations of biomarkers according to malnutrition risks classified by validated nutrition assessment tools. A total of 111 studies were included, representing 52,911 participants  from various clinical settings . The estimated BMI ( Malnutrition is highly prevalent in hospitalized patients yet remained underdiagnosed, especially in the frail, elderly population. Only 3 to 5% of hospitalized population is diagnosed with malnutrition, although it is estimated that 30–60% of the hospitalized population are malnourished ,2,3,4. DBecause of this complexity, practitioners have sought a rapid, more convenient laboratory means, usually involving serum biochemicals, measured as part of routine blood tests, to identify patients at risk of malnutrition. In clinical practice, blood diagnostics also have the advantage to ensure immediate nutrition assessment and prompt intervention for patients who are malnourished or at risk of malnutrition. Protein markers, such as albumin and prealbumin , were once the standard blood biomarkers for diagnosing malnutrition during hospital admissions. However, studies revealed that these are negative acute phase protein whose serum levels are affected not only by nutrition status, but also by many other factors such as inflammation, infection, liver damage, fluid status, etc. ,10,11. TThe correct interpretation of hematology test results and making accurate clinical decisions require well-defined reference values for the respective marker. However, there is still a lack of consensus on the optimal cutoffs and reference ranges of marker levels for evaluation of malnutrition risk among older people. Blood biomarker concentrations often vary with age, sex, race, diet, metabolism, and disease status . The appGiven that blood biomarkers for malnutrition remain widely used in clinical practice, we conducted a comprehensive systematic review and meta-analyses of blood biomarkers commonly used to assess nutrition status in older adults from various clinical settings, including hospitalized inpatients, outpatients, institutionalized elderly and those received home care. The purpose of this study is to summarize publications on blood biomarkers for malnutrition among order adults and to evaluate them against validated nutrition screening tools. We explored the reference ranges of potential biomarkers for subjects at different degree of malnutrition risk, while taking into account the effect of age, gender and disease acuteness. Although it is well-known that malnutrition and sarcopenia  may overlap, this review did not attempt to address biomarkers that associated with sarcopenia, mostly due to limitation on the number of malnutrition-related publications that collect body composition data along with strength/functionality data. This review will give an overview of the ability of potential blood biomarkers to identify malnutrition risk among patients with or without stress of acute disease. It provides a general guideline for using blood biomarkers to assess malnutrition that is relevant for subjects in various clinical settings.The review was planned and conducted following the UK National Health Service Centre for Reviews and Dissemination (CRD) guidance , and repA range of electronic databases were searched using the ProQuest Dialog (PQD) service, including MEDLINE (1950 to present), EMBASE (1974 to present), Embase, Foodline: SCIENCE, Food Science & Technology Abstracts (FSTA), and ProQuest Dissertations and Theses Professional. We also searched PubMed, Cochrane Database of Systematic Reviews and Cochrane Central Register of Controlled Trials. The search strategy for PQD is detailed in We included studies evaluating blood levels of biomarkers related to malnutrition in adults from various clinical settings, whether they live in the community or they are in hospital or institutionalized in a long-term care facility. Included studies were required to examine association of blood biomarkers to any of the following validated nutrition screening and assessment tools: Malnutrition Screening Tool (MST), MUST, NRS-2002, Geriatric Nutritional Risk Index (GNRI), Nutritional Risk Index (NRI), Instant Nutritional Assessment (INA), Nutrition Screening Initiative (NSI) and Short Nutritional Assessment Questionnaire (SNAQ), MNA, Mini Nutritional Assessment-Short Form (MNA-SF), SGA, and Detailed Nutritional Assessment (DNA). Studies specifically on patients with kidney disease or cancer were not considered. For studies with multiple publications on the same study population, we selected the study with the largest number of subjects or the most recent publication as the primary report.Titles and abstracts resulting from the search strategy were evaluated by two independent investigators. Disagreements were resolved by consensus and, when necessary, by consultation with another investigator. One reviewer performed the data extraction with a second investigator checking the data extraction forms for accuracy and completeness. The studies were grouped according to individual blood biomarkers and the validated nutrition screening tools used. Data were extracted and collated on the following study characteristics: reference, study design, clinical settings , subjects , health concerns, nutrition assessment tools, biomarkers, key results, and conclusions. Patients admitted to the intensive care unit (ICU), acute/critical/emergency units, experienced infections, trauma, burns, or post-surgery were considered acute. Disease acuteness was coded as ICU, acute (Y), non-acute (N) or both (B). The primary outcome required was mean and standard deviation (SD) of biomarker values in different malnutrition groups classified by a nutrition assessment tool. Where appropriate, we contacted lead authors for missing data and to clarify suspicious values of biomarker measurements. Outliers of biomarker values were flagged and checked during subsequent statistical analysis.The quality of included studies was assessed using the National Institutes of Health (NIH) Quality Assessment tool for Observational Cohort and Cross-Sectional Studies . The tooI2 statistic. An I2 > 50% represented substantial heterogeneity. A fixed-effects model with weighting of the studies was used when there was a lack of significant heterogeneity (p > 0.10), while a random-effects model with weighting of the studies was used when there was heterogeneity between studies (p ≤ 0.10). We assessed publication bias by visual inspection of the funnel plot and Egger’s test for funnel plot asymmetry. A sensitivity analysis was conducted to determine whether including patients with acute disease would substantially affect the biomarker levels. Two-sided P value was used with an α level of 0.05. all Statistical analyses were performed with R version 3.2.3  with the package metaphor, version 1.9-8 .Mixed-effects, multiple linear regression models for meta-analysis were used to compare between-group differences in mean biomarker levels according to malnutrition status defined by validated nutrition screening tools, while adjusting for the effect of age and gender proportion. The heterogeneity among studies was estimated by Q test and n = 39,027), outpatients (n = 4159), and those received health care provided at home (n = 3400) or in the community (n = 6325), of which 4071 are elderly institutionalized in various types of long-term care facilities including nursing home, retirement home, and group homes. As for nutrition assessment tools, MNA was the most frequently used tool for malnutrition assessment, followed by MNA-SF, SGA, GNRI, and NRI. Other tools, such as NRS-2002 and MUST [After screening 936 publications, 111 studies representing a pool of 52,911 participants  finally met the inclusion criteria. The selection process is illustrated in and MUST ,18,19, wand MUST , and NSIand MUST  were useA total of 43 blood biomarkers were identified in the literature reviewed but only 17 of them had 3 or more studies for meta-analysis. BMI was included as a reference for the analysis because most of the nutrition assessment tools include BMI as a criterion to define malnutrition. The most commonly studied blood biomarker was albumin, followed by hemoglobin (Hb), total cholesterol (Tch), total lymphocyte counts (TLC), prealbumin (PAB), C-reactive protein (CRP), total protein (TP), transferrin (TF), creatinine (Cre), triglycerides (TG), white blood cells (WBC), blood urea nitrogen (BUN), % hematocrit (HT), iron, and estimated glomerular filtration rate (eGFR). The remaining blood biomarkers were evaluated in no more than 3 studies for any of the nutrition assessment tools. In addition to blood biomarkers, BMI is still commonly used as part of the nutrition assessment and has been unanimously reported in almost all the studies. Therefore, it would be helpful to also include BMI as a reference biomarker for the meta-analysis.The overall quality of the included studies was rated as fair . All stup < 0.001, I2 > 50%) across studies for all the biomarkers. No publication bias was observed by evaluation of the funnel plots (p < 0.001), SGA (p < 0.001), and NRS-2002 (p < 0.01), but also those at low risk of malnutrition as determined by MNA (p < 0.01) and SGA (p < 0.01) , NRS-2002 (25.77 kg/m2) and GNRI (23.96 kg/m2)  . The est6 kg/m2) .p < 0.001), hemoglobin (p < 0.001), total cholesterol (p < 0.001), prealbumin (p < 0.001) and total protein (p < 0.05) for subjects identified by MNA as at high malnutrition risk were statistically lower than those without a malnutrition risk (p < 0.01) level (p < 0.001) and prealbumin (p < 0.05) were able to identify patients at risk of both high and low malnutrition as assessed by GNRI  level . Since G by GNRI .The estimated biomarker mean concentrations for each malnutrition risk groups also varied among different assessment tools, with the highest estimates observed for NRS-2002, followed by MNA and SGA. For example, the estimated mean albumin concentration for patients at high risk of malnutrition was 3.42 , 3.31 , and 3.08  g/dL by NRS-2002, MNA and SGA respectively . This inA sensitivity analysis indicates that some biomarkers, especially CRP and TLC, were dramatically affected by stress of acute illness . When stBlood biomarkers, especially albumin, are often used for diagnosis of malnutrition in clinical practice. Yet there is a lack of evidence-based clinical guidance to support their application under specific conditions and settings . The pre3/µL) and WBC (>8 × 103/µL) are within the normal range. While CRP, TLC, and WBC may serve as indicators of inflammatory status, they are not good markers of malnutrition status. As for other blood biomarkers, including transferrin, creatinine, total triglycerides, iron, and % hematocrit, we do not find sufficient evidence to support their use as a marker of malnutrition.Results from our meta-analysis showed that several blood biomarkers, including albumin, prealbumin, hemoglobin, total cholesterol, and total protein, are useful biochemical indicators of malnutrition, even with the presence of chronic inflammation. Inflammation, due to disease or ageing, is an important etiologic factor to the development of malnutrition . Since a2 for subjects 72 years of age is suggested, which is consistent to a recent ESPEN consensus that suggested using <22 kg/m2 in subjects older than 70 years [2 provided by World Health Organization (WHO) to define malnutrition would likely fail to identify some patients who are at risk of malnutrition.Although BMI as a biomarker is not the focus of our meta-analysis, it is worth presenting the results here given its widespread utilization in clinical practice, Low BMI is commonly accepted to be a criterion for malnutrition diagnosis . It is n70 years . A simplAlbumin is the most extensively studied proteins for diagnosing malnutrition. Our results support the role of serum albumin as a useful indicator of overall nutrition status for older adults in non-acutely ill states . HoweverSimilar to albumin, serum prealbumin has also been used as a blood marker of malnutrition. The normal value of prealbumin is 20–40 mg/dL and a mild degree of malnutrition is indicated by a range of 10–20 mg/dL, with serious malnutrition signaled by values below 10 mg/dL . While rThe recent etiology-based approach for diagnosis of adult malnutrition recognizes the importance of inflammation in the pathophysiology of malnutrition . In acutBlood hemoglobin, total cholesterol and total protein were also found to be useful markers of adult malnutrition. More importantly, it was observed that these biochemical measures were insensitive to acute disease stress. The normal hemoglobin range is generally defined as 13.5 to 17.5 g/dL of blood for men and 12.0 to 15.5 g/dL for women . The estCombining several biomarkers that are reflective of nutrition status may have the potential to increases sensitivity and specificity. Two studies attempted to combine validated screening tool(s) and/or a blood biomarker to screen malnutrition risk ,104. SevAlthough our findings are based on observational studies, there are several advantages. First, this is the first comprehensive meta-analysis conducted to assess the association between malnutrition risk and blood biomarkers in older adults. Our review covered a wide array of serological markers and provided their reference values by taking into account the potential confounding effect of age, gender, and acute disease stress. Second, a dose-response relationship was observed between biomarker levels and degree of malnutrition risks. Third, we used validated nutrition screening/assessment tools as the standard against which to evaluate the potential of individual blood biomarkers to detect malnutrition risk. We were able to compare and cross-validate results using different tools, including MNA, SGA, NRS-2002, GNRI and MNA-SF. Given that blood biomarkers are widely used in clinical practice, our findings provide useful evidence-based clinical guidance to support their application under specific conditions and settings.However, there are some limitations that warrant consideration. First, the sample size for each individual marker is unequal. Albumin is the most commonly used blood biomarker for assessing malnutrition, followed by hemoglobin, total cholesterol, total lymphocyte counts, prealbumin, and total protein. Fewer studies evaluated the other blood biomarkers, including transferrin, creatinine, total triglycerides, iron, and % hematocrit, so that the statistical power may be limited. Further investigation with greater numbers would be required to establish whether these blood biomarkers are valid indicators for risk of malnutrition.Second, our result is based on blood biomarker concentrations measured in only one point of time, either at admission to the hospital, visited a health care provider, or during the pre-surgery examination. Therefore, the diagnostic performance of the individual biomarker is considered an indicator of the general nutrition status at the time when blood sample was taken. A large prospective study with well-controlled time points for blood collection is needed to truly validate these findings. In addition, analysis of the biomarkers on a single diagnostic platform may help determine if the observed variability in the cut points for the biomarkers are truly clinical variability or variations due to differences in the diagnostic assays used across the studies. It would also be useful to determine if the identified markers can change in response to nutrition repletion, which can then be useful markers to determine early benefits of an intervention for malnutrition.Third, while our study encompassed a large number of subjects from various clinical setting , studies specifically on patients with kidney disease or cancer are not included. Such a selection better reflects the general hospitalized/institutionalized population. It also recognizes the fact that many blood biomarkers are affected by kidney malfunction and cancer pathology.Finally, we would like to point out that although our results provide a useful guideline for using blood biomarkers to assess malnutrition in clinical practice, we do not encourage the sole use of a single biomarker to diagnose malnutrition in a patient. Blood biomarkers should only be used as a complement to a thorough physical examination. A medical diagnosis can only be made through careful consideration of findings from all available information including but not limited to medical and dietary records, physical examination, and laboratory tests.In conclusion, this systematic review confirmed that BMI and several blood biochemicals, including albumin, prealbumin, hemoglobin, total cholesterol, and total protein, are useful biomarkers for adult malnutrition, even with the presence of chronic inflammation. However, the reference ranges and cut-offs may need to be updated to avoid under-diagnosis of malnutrition. It confirms that in acute health care settings, albumin and prealbumin must be carefully interpreted because they may be affected by changes brought about by acute disease and the associated inflammation. Future studies should address the important questions of whether these blood biomarker response to nutrition treatment and which biomarker works better. Additionally, efforts can be devoted to develop an algorithm using factors that together are most predictive of risk of malnutrition. In older people, malnutrition and sarcopenia often coexists. Further research is also needed to determine the association of malnutrition biomarkers with sarcopenia and that of other anthropometric measurements such as body muscle mass, mid-arm circumference, calf circumference etc. with the nutrition assessment tools.