Human observers perceptually discriminate the dynamic deformation of materials in the real world. However, the psychophysical and neural mechanisms responsible for the perception of dynamic deformation have not been fully elucidated. By using a deforming bar as the stimulus, we showed that the spatial frequency of deformation was a critical determinant of deformation perception. Simulating the response of direction-selective units  to stimuli, we found that the perception of dynamic deformation was well explained by assuming a higher-order mechanism monitoring the spatial pattern of direction responses. Our model with the higher-order mechanism also successfully explained the appearance of a visual illusion wherein a static bar apparently deforms against a tilted drifting grating. In particular, it was the lower spatial frequencies in this pattern that strongly contributed to the deformation perception. Finally, by manipulating the luminance of the static bar, we observed that the mechanism for the illusory deformation was more sensitive to luminance than contrast cues. From the psychophysical and computational points of view, the present study tried to answer the question, “how do human observers see deformation?”. In the psychophysical experiment, we used a clip wherein a bar dynamically deformed. We also tested the illusory deformation of a bar, which was caused by tilted drifting grating, because it was unclear whether the illusory deformation could be described by our model. In the computational analysis, to explain psychophysical data for deformation perception, it was necessary to assume an additional unit monitoring the spatial pattern of direction responses of MT cells that were sensitive to local image motion.Materials in the real world are often non-rigid. The material non-rigidity dynamically produces the deformations of contours and textures in the retinal images. The dynamic deformation of retinal images is a rich source of visual information allowing the visual system to assess material properties in the real world. For example, from the dynamic deformation of retinal images, human observers can recognize transparent liquid , transpaImportantly, however, the visual mechanism for detecting dynamic deformation itself has not been thoroughly examined. Previous studies have reported that observers reported dynamic deformation when local motion integration did not provide evidence for rigid motion ,b at a lThe purpose of this study was to psychophysically and computationally specify the mechanism that underlies the perception of dynamic deformation. In experiment 1, using stimuli with a physically deforming bar we show that the spatial frequency of deformation is an important factor to phenomenally determine deformation perception. By simulating both spatiotemporal energy responses at the V1 level and the responses of direction-selective units  participated in this experiment. All observers in this study reported having normal or corrected-to-normal visual acuity. They were recruited from outside the laboratory and received payment for their participation. Ethical approval for this study was obtained from the ethics committee at Nippon Telegraph and Telephone Corporation . The experiments were conducted according to principles that have their origin in the Helsinki Declaration. Written, informed consent was obtained from all observers in this study.Stimuli were presented on a 21-inch iMac (Apple Inc. USA) with a resolution of 1280 × 720 pixels and a refresh rate of 60 Hz. A colorimeter  was used to measure the luminance emitted from the display. A computer  controlled stimulus presentation, and data were collected with PsychoPy v1.83 , 2009.2). The luminance of the background was 76 cd/m2.In the stimulus , the edgMovie 1.Some examples of stimulus video clips as used in Experiment 1.10.1523/ENEURO.0278-19.2020.video.1Each observer was tested in a lit chamber. The observers sat 102 cm from the display. With each trial, a stimulus clip having a deforming bar was presented for 3 s. After the disappearance of the clip, a two-dimensional white noise pattern (with each cell subtending 0.16° × 0.16°) was presented until the observer’s response. The task of the observers was to judge whether a bar dynamically deformed or not. The judgment was delivered by pressing one of the assigned keys. Each observer had two sessions, each consisting of seven spatial frequencies of the modulation × 10 repetitions. Within each session the order of trials was pseudo-randomized. Thus, each observer had 140 trials in total. It took ∼20 min for each observer to complete all sessions.Twelve people  participated in this experiment. Their mean age was 38.2 . Although seven of them also already participated in the previous experiment, none was aware of the specific purpose of the experiment because there was no preliminary explanation or debriefing provided.Apparatus was identical to that used in experiment 1.2). The orientation of the background grating was selected from the following seven levels . The spatial frequency was selected from the following 3 levels . Drift temporal frequency was kept constant at 1 Hz. The drift direction was randomly determined. The luminance contrast of the grating was set at 0.75, and thus the luminance level of the grating ranged between 37 and 112 cd/m2. The drifting grating was windowed by a horizontal Gaussian envelope with a SD of 0.62°.As shown in Movie 2.Some examples of stimulus video clips as used in the 25.8 cpd condition of Experiment 2.10.1523/ENEURO.0278-19.2020.video.2Movie 3.Some examples of stimulus video clips as used in the 12.9 cpd condition of Experiment 2.10.1523/ENEURO.0278-19.2020.video.3Movie 4.Some examples of stimulus video clips as used in the 6.4 cpd condition of Experiment 2.10.1523/ENEURO.0278-19.2020.video.4Procedure was identical to that in the previous experiment except for the following. With each trial, a stimulus clip having a static bar and drifting grating was presented for 3 s. After the disappearance of the clip, visual white noise (each cell subtending 0.16° × 0.16°) was presented until the observer’s response. The task of the observers was to judge whether the static bar dynamically deformed or not. Each observer had four sessions, each consisting of three spatial frequencies × seven orientations × five repetitions. Within each session the order of trials was pseudo-randomized. Thus, each observer had 420 trials in total. It took 30–40 min for each observer to complete all four sessions.Twelve people who had participated in experiment 1 again participated in this experiment. Still, none was aware of the specific purpose of the experiment.Apparatus was identical to that used in experiment 1.2 wherein 76 cd/m2 was the luminance of a neutral gray level).Stimuli were identical to those used in experiment 1 except for the following. The background grating orientation was 1° or 16°, which respectively produced strong deformation and non-deformation responses in experiment 1. The grating spatial frequency was kept constant at 12.9 cpd. As shown in Extended Data Movie 5.Some examples of stimulus video clips as used in Experiment 3.10.1523/ENEURO.0278-19.2020.video.5Procedure was identical to that used in experiment 1 except for the following. Each observer had four sessions, each consisting of two levels of grating orientation × nine luminance levels of the bar × five repetitions. Within each session the order of trials was pseudo-randomized. Thus, each observer performed 360 trials in total. It took ∼20 min for each observer to complete all of both sessions.For the stimuli that were used in experiments 1 and 2, we simulated the responses of direction-selective units on the basis of previous studies . We extrIn experiment 1, we set the width of the extracted area for analysis at 0.08° because the amplitude of contour deformation applied to the bar was 0.04°. In experiment 2, based on the spatial frequency of the background grating, we changed the width of the extracted area for analysis . The height of the extracted area was constant at 4.97°. The extracted area was first analyzed by a set of spatiotemporal filters. In experiment 1, width, height, and spatial wavelength of the spatiotemporal filters were all 0.08°. In experiment 2, width, height, and spatial wavelength of the spatiotemporal filters were also consistent with the width of the extracted area, that is, 0.16°, 0.0 °, and 0.04° for 6.4, 12.9, and 24.8 cpd conditions, respectively. The filters with different phases (0π or 1.0π) were independently applied to stimuli and the outputs of the filters with different phases were later summed after the half-wave rectification and normalization described below. The number of filter orientations was 24 . The position of the filters did not spatially overlap. In experiment 1, 64 filters covered the extracted area. In experiment 2, 32, 64, and 128 filters covered the extracted area for 6.4, 12.9, and 24.8 cpd conditions, respectively.In both experiments, the temporal size of the filter was 6 frames (for ∼100 ms) and the temporal frequency was fixed at 0.4 Hz. The combination of temporal and spatial frequencies of the filter was optimized to the stimulus speed in the stimuli. We adopted the temporal properties because we wanted to extract a one-way modulation of the moiré pattern.The responses of the filters were half-wave rectified and normalized as reported in the previous study . In the The pooled responses were filtered by a direction-tuned filter , 2014 thf denotes spatial frequency, ϕ denotes phase, and x denotes spatial position. D was defined by the following formula, θ denotes the motion direction and ranges from 0 to 2π, and α denotes the preferred direction of the kernel. Here, α was set to 0° (leftward direction) on the basis of the spatial pattern of the direction-selective units as shown in The following analysis was conducted to calculate the normalized cross-correlation (NCC) between spatiotemporal motion energy (or the response of direction-selective units) and the kernel of assumed higher-order units that are possibly sensitive to the spatial variation of spatiotemporal motion energy or the responses of direction-selective units. The kernel was defined by the product of the spatially sinusoidal pattern S and the directionally sinusoidal pattern D A. S was K is the kernel, I is the spatiotemporal motion energy (or the pattern of direction-selective units), and Kμ and Iμ are the mean of K and I, respectively. This NCC is often called as the zero-mean NCC. The f of (1) took one of the following levels: 0.1, 0.2, 0.4, 0.8, 1.6, 3,2 and 6.4 cpd. The ϕ was tested in 32 steps (each step = 0.0625 π), and the maximum value among the 32 outcomes based on the 32 steps was considered to be the NCC of the kernel.To see how the kernel A matchedThe purpose of experiment 1 was to specify psychophysical parameters to cause the perception of dynamic deformation and look into a possible relationship between the psychophysical data and the simulation data of neural responses such as V1 motion energy and MT pattern motion cells. Although a previous study  investigF = 38.358, p < 0.0001, 2pη = 0.86]. Multiple comparison showed that the proportions in the 0.1, 0.2, 0.4, and 0.8 cpd conditions were significantly higher than the proportions in the 1.6, 3.2, and 6.4 cpd conditions (p < 0.05). The results showed that the lower spatial frequency of deformation contributed to the perception of dynamic deformation more strongly than the higher spatial frequency of deformation, consistent with the previous study  of the fitting is plotted in Next, we fitted an exponential function to the psychophysical data as a function of the NCCs. The coefficient of determination  = 27.440, p < 0.0001, 2pη = 0.72]. The main effect of the grating orientation was also significant . Interaction between the two factors was significant . The main effect of the background grating orientation was also significant . Interaction between the two factors was also significant . The simple main effect based on the significant interaction showed that the proportion for 76 cd/m2 was significantly lower than the proportions for other luminance conditions when the background orientation was 1° (p < 0.05). The results showed that the deformation perception was attenuated when the bar luminance was neutral gray, suggesting that the mechanism for the deformation perception in our illusion is most sensitive when there is a difference in overall luminance between a bar and its background. Moreover, the illusory deformation was still reported even when the luminance of the bar itself was outside of the luminance contrast range of the background grating. The results suggest that the deformation illusion triggered by moiré patterns occurs with a flexible relationship between the bar and the background grating.In This study investigated the mechanism responsible for deformation perception by human observers. Our data suggest that the brain determines deformation perception on the basis of the spatial pattern of the responses of the direction-selective units. Moreover, it was shown that the mechanism underlying the deformation perception was more selective to luminance-defined than contrast-defined features.Because the spatiotemporal motion energy did not well explain the deformation perception in our stimuli, we believe that the deformation perception is not based on a series of static deformations. Rather, it seems plausible to assume that motion mechanisms solving the aperture problem produce the kind of dynamic deformations, as reported in the previous study . BesidesPrevious studies have reported that the human visual system is sensitive to direction-defined stripes ,b and/orWhat is the possible neural mechanism for seeing deformation? It is well known that velocities are processed in the MT area. In general, velocities are captured via a large receptive field and hence processed globally . On the In this study, we did not closely investigate the role of speed in the perception of dynamic deformation. Specifically, we assessed only direction parameters in the computation model, while the model proposed in previous studies  could asHow is the illusory deformation in our stimuli related to the footstep/inchworm illusion? In the footstep illusion , an objeWe acknowledge that the results of experiment 3 are explained in terms of the act of spatiotemporal filters at the first stage of our model. To detect contrast-defined features, we need to assume nonlinear processing before evaluating the spatiotemporal structure of contrast modulation. The spatiotemporal filter does not, in general, follow a nonlinear processing as like a rectification and hence it is impossible for the simple spatiotemporal filters to detect the contrast-defined structures. In experiment 3, the deformation perception was attenuated when the overall luminance of the background was equivalent to a bar. The results are consistent with the interpretation that the illusory deformation perception occurs when the spatiotemporal filters that are sensitive to luminance structures properly detect the spatiotemporal structures that eventually causes the activation of the direction selective units producing the lower spatial frequency pattern. On the other hand, we would like to emphasize that after the detection of spatiotemporal luminance structure the brain solves an aperture problem to determine global motion directions when deformation is dynamic. The solution to the aperture problem is mediated by the filters at MT. Thus, we need to assume the two-stage processing to entirely describe the dynamic deformation perception in our stimuli.In this study, we focused on shearing deformation but not on compressive deformation. Both shearing, and compressive deformations exist in the image deformation of natural materials such as the flow of a transparent liquid. Showing that the sensitivity to the compressive deformation was higher than the sensitivity to the shearing deformation,


Malus domestica Borkh.) cultivar Rubin was grafted on dwarfing rootstocks P60, planted in single rows spaced 1.25 m apart with 3.5 m between rows. In contrast to plant senescing reflectance index and nitrogen balance index, the photochemical reflectance index was significantly higher in 2018 compared with 2017. Such differences might be caused by drought stress on the summer and fast recovery before harvest time when measurements were made. The movement of nutrients and water disrupted by trunk incision had significantly negative effect on reflectance indices regardless on the year. Mechanical pruning with trunk incision and calcium-prohexadione lead to decreased dry to fresh weight ratio by 10–12% in first year of treatment. Mechanical pruning had significantly negative impact on photosynthetic rate, compared to pruning by super spindle it decreased 47%. Strong positive correlation between PRI and NBI 0,89–0,94, and strong negative correlations between PRI, NBI and PSRI −0.88 – (−0.91) were determined.The aim was to find out the impact of stress, caused by agrotechnological tools on photosynthetic behaviour of apple trees. The apple tree ( From 100% sun light only 40% is typical light interception by orchard systems and 15% (out of 40%) is limited by orchard design and leaf area2. During photosynthesis, solar energy binds to dry matter, so the photosynthetic behavior of the plant can be evaluated. Increasing photosynthesis goes to increase in biomass. Optimal plant photosynthesis system work is the main part of the biomass quantity3.Not only environmental factors but also plant canopy ant light penetration into canopy strongly influence photosynthetic productivity and plants’ growth. Photosynthesis productivity depends on light, water, CO5. Nitrogen (N) is one of the most important components of photosynthetic productivity is the also, it is a part of chlorophyll and it is one of the most important elements for biomass production6. The chlorophyll index (Chl) and the nitrogen balance index (NBI) measure the relative amount of chlorophyll and nitrogen at the same point on the leaf in the same moment and it may convey N dynamics in apple trees systems8. Plant synthesizes proteins and favors primary metabolism(nitrogen containing molecules) containing chlorophyll and just a few flavonols is shown by higher nitrogen balance index. NBI status of leaves has a relationship to light availability and light limitation as shown in literature9.Non-destructive measurement methods easily measure variable optical properties without affecting the leaf at various stages of its development. In one measurement several functionally relevant leaf parameters is done without interruption10. PRI is commonly used as an appropriate indicator of abiotic factors that limit photosynthesis11. PRI via the xanthophyll cycle is related with photosystem II (PSII) and it is using as an alternative for light use efficiency. Photochemical reflectance index was applied as an active probe of pigment conversion, it can reflect almost all changes in the pigment content of the xanthophyll cycle12. Other authors have investigated that changes in pigment content and ratio due to leaf growth, aging, or chronic stress play an important role along with xanthophyll pigment epoxidation in the PRI signal. The photochemical reflectance index responds to the chlorophyll / carotenoid ratio in leaves due to species, age, and environmental conditions13. While PRI characterizes the photosynthetic efficiency, the plant senescing reflectance index (PSRI) is sensitive to the changes of carotenoids and chlorophyll ratio and it was used as a quantitative measure of leaf senescence. They are both commonly used to describe the changing physiological state of vegetation15. Reflectance spectra in the visible and near infra‐red range of the spectrum, acquired for maple (Acer platanoides L.), chestnut (Aesculus hippocastanum L.), potato (Solanum tuberosum L.), coleus (Coleus blumei Benth.), leaves and lemon (Citrus limon L.) and apple  were studied and it was found that leaf senescence effect is depending on pigment composition14.The Photochemical Reflectance Index (PRI), which is based on leaf reflectance at 531 nm, is suitable for tracking variations in photosynthetic activity from leaf16. Alarcon and Sassenrath17 showed cotton leaf photosynthesis efficient dependence from shapes and canopy architecture. This shows that the effectiveness of photosynthesis can be controlled through the shape of the canopy and the best results were based on the high number of small-to-medium sized leaves. Like all plants, the ability of apple trees to sustain higher photosynthetic rates (PN) depends on the whole tree canopy and its regulation18. Also, Zhang et al.19 showed that photosynthesis consider from temperature, CO2 and especially light intensity and chlorophyll content. Moreover, photosynthetic rates are sensitive to various stresses like salt-alkaline, salt, water deficiency, shading and other20. One of the reasons for the PN reduction is the stress of water drought and Ohashi et al. measured the decrease by up to 34%21. The inhibition of photosynthetic rate under water stress may be attributed not only to stomatal closure but also on several biochemical and photochemical processes23.Photosynthesis activity is associated not only with the yield of individual leaves, but also with the optimal use of light, considering the penetration of light through the canopyet al.24 showed that heavy crop load decreased fruit size and weight. Denser canopy also has negative impact on fruit mass and yield25. Although calcium-prohexadione is using for smaller terminal growth it also has the impact for fruit weight. Usually fruit weight decreases by use of calcium-prohexadion. As Privé et al.26 showen Calcium-prohexadion can increase light penetration into the inner canopy of fully-grown apple trees and thus improve flower bud formation and expected yield.‘Rubin’ is large fruit variety. De Salvador 29. Trunk girdling not only increase fruit yield, but also can increase the influence of other agrotechnical tools using foliar spray or fertilization32.Water and nutrients movement are disturbed by trunk girdling as well as by trunk incision. Trunk girdling and ringing significantly increase yield but most of the studies shown that photosynthetic behavior decreases in the same time as yields increaseManual pruning of orchards requires a lot of resources and manpower to reduce the cost of gardening through mechanical pruning, trunk cutting, chemicals and more. Despite the amount of research on the relationship between the optical properties of apple leaves and the rate of photosynthesis in different apple agrotechnology tools, there is little. Thus, the aim was to elucidate the influence of stress caused by light penetration on canopy and agro-technological measures on photosynthetic behavior of apple trees.Malus domestica Borkh.) cultivar Rubin was grafted on dwarfing rootstocks P60. Apple trees were planted in 2010 year in single rows spaced 1.25 m apart with 3.5 m between rows. Pest and disease management was carried out according to integrated plant protection practices and orchard was not irrigated. Soil conditions of the experimental orchard were following: clay loam, pH 7.3, humus 2.8%, P2O5 255 mg kg−1, K2O 230 mg kg−1. Three single trees were fully randomized. The non-destructive and biometric analysis were performed in harvest time in the end of September. The samples were taken from five leaves in two levels: at 1.0–1.2 m and 1.8–2.0 m above the ground. Five agrotechnological tools were used: (1) each year hand pruning forming slender spindle (control); (2) mechanical pruning (each year) with hand pruning every second year; (3) mechanical pruning (each year); (4) trunk incision using chain saw +mechanical pruning (each year); (5) mechanical pruning (each year) +spraying with calcium-prohexadione. Mechanical pruning goes with hand pruning ones in three years removing the oldest branches from the stem base. Reducing the movement of water and nutrients by cutting the trunk also suppresses tree growth, reduces the number of shoots and at the same time increases the lighting of the canopy at the top of the it. Mechanical pruning, on the other hand, only shortens the branches, without the use of manual pruning, increasing the number of large branches, which increases the darkening of the crown. Also, trees grow actively in the upper part of the canopy, densify and multiply branches, which cast heavy shadows on the entire tree. Using only mechanical pruning, the trees resemble a hedge. While in the case of manual pruning, the large branches are removed from the trunk, there are no large branches at the top of the canopy, and the canopy is easily lightened and does not darken the rest of the crown.A field experiment was carried out in an intensive orchard. The apple tree  from 9 to 12 am. Reflection spectra obtained from the leaves were used to calculate photochemical reflectance index (PRI) and plant senescing reflectance index (PSRI). PRI shows changes in the xanthophyll cycle, using the following formula:678, R500 and R570 represent the leaf reflectance integrated over a 10 nm wavelength band centred on 678, 500 and 570 nm, respectivelyPSRI was proposed as being sensitive to the senescence phase of plant development and it’s calculated by formula:NBI was evaluated using no-destructive measurement of leaf chlorophyll and flavonoid content  in the epidermis .2] (400 μmol mol−1), light intensity (1000 μmol m−2 s−1) and the flow rate of gas pump (500 mmol s −1) were set.Photosynthetic rate was determined at 9:00–12:00 am by using a LI-6400XT portable open flow gas exchange system . Reference air [CO2), twenty leaves were randomly sampled from all tree canopy and measured with a leaf area meter . The dry mass of twenty leaves was determined by drying apple tree leaves in 70 °C for 48 h.  to constant weight33.To determine the leaf area , The data were processed using two-way ant three-way analysis of variance (ANOVA) Analysis of variance was performed with Duncan’s least significant difference test (P < 0.05).Mechanical pruning results denser crown and light penetration into canopy. The lower light penetration into the crown resulted the decrease of PRI (25–100%) Fig. . HoweverNitrogen balance index showed significant differences only in the lower part of canopy. Mechanical pruning with manual pruning increased NBI by 38% compared to control. Meanwhile, using trunk cut, the NBI variation is insignificant compared to the control Fig. Figure 3NN). PN decreased by 47% using only mechanical pruning compared to pruning by super spindle. Mechanical pruning with hand pruning, trunk incision and calcium-prohexadione also resulted in significant decrease of PN 10–20% compared to control.Trunk incision and calcium-prohexadione significantly decreased DW/FW ratio to 10–12% Fig. . Compare−1 yield in 2018 . The first two components accounted for the majority of variation in the data set (88.59%). The variation of 82.60% could be explained by the first principal component (PC1) and 5.99% by the droughty principal component (PC2). Strong positive correlation between PRI and NBI 0,89–0.94, and strong negative correlations between PRI, NBI and PSRI −0.88 – (−0.91) were found , light intensity, but also the effect of temperature on PRI (PRI and PSII efficiency). In their research is shown that at lower temperatures and dark green leaves, PRI values increase. Similarly, Weng et al.35 found that the relationship between the efficiency of PRS II and PRI varies with temperature and leaf color, they argue that lower PRI values are associated with increased illumination of mango tree leaves. While getting into the details of other authors’ data, and considering that this research comes from autumn measurements, when daytime temperatures drop, we find that during mechanical pruning, PRI responds more to temperature than to light intensity  organs by response to drought. We see that NBI increased significantly only in the sunny and droughty years 43. However, Kozlowski and Pallardy44 observations of woody plants adaptation to drought stress showed that trees have developed mechanisms to deal with dehydration conditions that are different from herbaceous plants. The amount of unstructured carbohydrates may also reflect tree drought, which may lead to a decrease in the concentration of unstructured carbohydrates, but as Gaul et al. In45, this usually only occurs in the later stages of drought.Water deficit is one of the greatest limiting factor for photosynthesis and biomass accumulation47. Trees that have been touched after a short drought have no major changes, even some tree species are able to recover quickly after a drought49. During treatment years was one drought episode form May till July , then another drought episode from the end of August till harvest time appeared. Apple trees were affected by short-term drought in 2018 but form the PRI and PSRI values it can be stated that the trees quickly recovered and functioned even better than in normal years.Water intake increases with increased root fine growth, deep root formation, and solvent accumulation to reduce the water potential in the root tissue. Only then does the drought level become too severe, and the plants react by activating the mechanisms that protect tissues from cell damage, mainly by the action of protective solutes and proteinsN is up to 50% lower in shade leaves compared to sun leaves50. Negative shade impact for trees photosynthesis also was described in several research like for Acer rubrum and Betula papyrifera51, Fagus syltica52, Ginkgo biloba53, Malus domestica54. However, shading can have positive impact for photosynthesis when it goes with drought but in our research there was no significant impact of shading55.Denser canopy formed by mechanical pruning decreases another limiting factor – light penetration into canopy. P56 that calcium-prohexadione had no significant impact for fruit weight for apple cultivar ‘Rubin’ . Mechanical pruning increased apple yield upto 45% but decreased fruit size by 4% in two years average. In general, the recommendation for industrial orchards is hand pruning every second year, mechanical pruning combined with reduced manual pruning lowered orchard maintenance but retained apple yield and photosynthetic activity.Strong positive correlation between PRI and NBI 0,89–0,94, and strong negative correlations between PRI, NBI and PSRI −0.88 – (−0.91) were found. Trunk incision disturbed nutrients and water movement from roots to canopy that resulted the significant decrease of photochemical reflectance index, photosynthetic rate, fruit mass and accelerated leaf aging. The lower light penetration in denser canopy formed using mechanical pruning also resulted in significant decrease of photosynthetic indices (PRI, PSRI, PDataset 1.


In this paper, we present a novel pipeline to simultaneously estimate and manipulate the deformation of an object using only force sensing and an FEM model. The pipeline is composed of a sensor model, a deformation model and a pose controller. The sensor model computes the contact forces that are used as input to the deformation model which updates the volumetric mesh of a manipulated object. The controller then deforms the object such that a given pose on the mesh reaches a desired pose. The proposed approach is thoroughly evaluated in real experiments using a robot manipulator and a force-torque sensor to show its accuracy in estimating and manipulating deformations without the use of vision sensors. As deformable objects are ubiquitous in many industries, automating their manipulation would have a great social impact. For instance, robots could perform tasks that are either dangerous or monotonous for workers. Examples of manipulation of deformable objects can be found in the automobile and aerospace industries, where cables and wires must be connected in order to assemble motors; in health care, where clothes are handled to dress disabled people; and in the food industry where meat and produce have to be processed with care. This has sparked interesting robotic solutions that attempt to address issues such as automating the manufacture of motors by manipulating cables as proposed by Rambow et al. , RousselIn this paper we propose an approach to both sense and manipulate the deformation of soft objectFollowing this introduction we review the state of the art for robotic sensing and manipulation of deformable objects in section 2. In section 3, we detailed our approach to estimate the deformation of an object while being manipulated as well as controlling its deformation. An experimental validation of the proposed approach is presented in section 4 followed by a discussion of the experimental results in section 5. Finally, our concluding remarks and future work are summarized in section 6.Two decades ago, one of the first works focusing on controlling the shape of a deformable object was presented by Wada et al. , where tdeformation feature vectors, based on a set of points tracked using markers. The proposed deformation vectors were defined by a set of points in an object to control a point, a distance or angle between two points, and a curvature described by three points. These approaches rely on a deformation Jacobian, which here refers to a matrix mapping the motion of the grippers to the deformation of the object. In Navarro-Alarcon et al.  behave rigidly and the farther the controlled points are from manipulated points the less rigid they become.Instead of relying on vision, Berenson  used simA novel approach for controlling the shape of an object without requiring an ISP formulation, was recently proposed in Navarro-Alarcon and Liu , where tOther works have focused on sensing the deformation of an object rather than controlling its shape using vision, either by relying on a mesh model as proposed in Petit et al. , Fugl etAs described in the previous section, most work concerning deformation control has either relied on vision sensing or assumed that the object's shape  is known at any time. However, vision systems are sensitive to variations in illumination, have difficulties segmenting objects with similar colors to their backgrounds and, perhaps most importantly, they are limited by occlusions arising while manipulating objects. In order to provide a complementary sensing modality, we propose a method using force sensing, coupled with a deformation model, to estimate the shape of an object while being deformed.Figure 3, which uses an improved version of the deformation sensing pipeline proposed in Sanchez et al. .The contact locations can be assumed to remain fixed.To estimate the deformation of the object we improve the sensing pipeline proposed in our previous work . As shown in Since the deformation sensing pipeline requires the input force to be described w.r.t. the object frame, we transform the force using the adjoint matrix p1, p2, p3) from the mesh, we can extract a pose as follows:pi is the position of the node at the i−th index of the mesh q and n represents the normal of the plane formed by the points. The position and orientation of the pose are given by the centroid c and the quaternion, respectively, where s is the scalar part of the quaternion and v is the vector part. To manipulate the mesh to a desired pose, let xc denote the current pose, as extracted by the method described above, and xd the desired pose for the mesh to reach. Then, we can define an error signal as follows:e into a twist command by multiplying it by a diagonal gain matrix Λ. As the twist is expressed in Cartesian space, we compute the joint velocities to command a robotic arm by joint velocity control as:J+ is the Moore–Penrose inverse Jacobian used for redundant manipulators.Since controlling the complete shape of an object  is a heavily underactuated problem, except for trivial cases, we reduce the control problem by focusing on controlling a single pose on the mesh. To this end, we first must extract a suitable pose from the mesh and consequently control that pose such that it reaches a target pose. To extract a pose we exploit the fact that the nodes of the mesh are connected by tetrahedra so three suitable nodes would form a triangle as shown in We evaluate the performance of our proposed approach by deforming four objects with different shapes and material properties. The two shapes of the objectshard (HR 45), medium (Bultex 30), and soft (Bultex 26) . For the test, the object was placed inside of a press that can be programmed to move on a vertical axis until a desired height and is equipped with a force sensor. The press was set to first move down to approximately compress the object 36 mm, then to move upwards until the compression was around 4 mm and finally the press was moved down until 44 mm of compression were reached. The results of this test can be seen in We attempted to estimate the material properties of a cube object with three different materials: It can be seen from these figures, that the behavior of all three materials is not only non-linear but also presents hysteresis. Hence, modeling these materials as linear will result in inaccuracies. Nevertheless, we can use the test findings to obtain a range estimation of the Young's modulus for each material and then tune them individually until the simulation behavior better matches reality.Two sets of experiments were conducted: one to measure the accuracy of the deformation sensing and the second one to assess the manipulation capabilities.The experimental setup used for both sets of experiments is shown in The ANSYSTo evaluate the sensing accuracy of our proposed pipeline we commanded a set of poses in the XZ plane of the robot frame, as shown in For each object, seven trials were performed consisting in three and six target positions for the block and the bar objects, respectively. The results are summarized in bar soft and block hard objects. For the bar object we set poses along the X and Z axis and the XZ plane as shown in The performance of manipulating deformation was evaluated on the Note that these control tasks could be solved by simply commanding a desired pose for the end-effector without regarding the mesh deformation. However, the purpose of this evaluation is to show that the shape of the object is estimated accurately enough to apply a controller.X and Z estimation of bar hard object are relatively small (< 2 cm) for 50% of the measurements, X and 2 cm in Z, however the precision is < 0.8 cm in X and < 0.3 cm in Z. Concerning the box plots of bar soft, for 50% of the measurements, they are < 2 cm as well in X and Z while the precision is < 0.4 cm in X and < 0.6 cm in Z. Concerning the blocks, the box plots are very small (< 2 mm) in Z for the block hard and 2.25 cm for the block soft. The variances in the blocks position estimation measurements are smaller than the ones of the bars as the blocks are stiffer, hence they are more resistant to the small force reading errors. Regarding the different position accuracy measurements for the objects used, it is dependent on how good are the physical parameters in Before using the deformation sensing model in the control pipeline in X and Z axes, against the response positions extracted from the mesh (red) and the reference positions extracted using the robot kinematics (blue). The figures show that the controller is able to bring the mesh to the desired positions, along the X and Z axes, within a 2 cm error  as shown in Command: Since the commanded signal is set by a GUI, the target poses for the controller result in a sharp slope.Elasticity parameters: The deformation model is highly sensitive to the elasticity parameters, namely, the Young modulus and Poisson ration. Thus, an adequate identification is necessary.Sensor noise: In order to obtain a smoother signal from the force sensors, a filter was applied but this introduced additional delay.The current pose estimation using the mesh is evaluated in the velocity control loop shown in the block diagram in These reasons cause the controller to have a response similar to damped step response and takes around 20 s to converge to the commanded goal. Hence, the error increases the most when a command is received, then it converges to a value close zero see .N), that's why the noises from residual non-contact forces and sensor noise have this much effect on the deformation model. For harder objects we expect the mesh to be less sensible to this noise.Concerning the small vibrations in the positions obtained from the mesh, they are mainly because of the small noise in the force readings. We note here that bar used in the tests is very soft  that it Z axes. Z are reached .The same controller shown in We have presented a pipeline that is able to simultaneously estimate and manipulate deformable objects without the need of a vision system, which is widely required by other approaches. The proposed deformation sensing pipeline takes as input a contact force and an initial mesh with the physical characteristics of the object. An FEM simulation estimates the deformation of the mesh based on the input force and in order to control the mesh, a pose, extracted from a given set of nodes of the mesh, is driven by a robot manipulator to reach a desired pose. The approach was experimentally validated using a robotic arm with a dexterous hand as an end-effector and an attached force-torque sensors. The experimental validation showed promising results and precision to a certain limit.The accuracy of the approach is limited due to several factors such as the control loop being closed on the pose extracted from the mesh which is subject to drift due to noise of the force-torque sensor readings. Additionally, the physical parameters of each object are not obtained in a straight forward manner, thus affecting the estimation between the deformation of the real object and the estimation by the model. In future work, we will focus on incorporating vision to correct the errors in the estimation of the deformation, as well as investigating the performance of different controllers for deformation control found in the literature (e.g., Navarro-Alarcon et al., All datasets generated for this study are included in the article/supplementary material.All the authors participated in the development and validation of the research work presented in this paper.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


We show that the PHY domain does not affect the initial conformational dynamics of the chromophore. However, it significantly accelerates the consecutively induced reorganizational dynamics of the protein, necessary for the progression of the photoisomerization. Consequently, the PHY domain keeps the bilin and its binding pocket in a more reactive conformation, which decreases the extent of protein reorganization required for the chromophore isomerization. Thereby, less energy is lost along nonproductive reaction pathways, resulting in increased efficiency.Phytochrome photoreceptors operate via photoisomerization of a bound bilin chromophore. Their typical architecture consists of GAF, PAS and PHY domains. Knotless phytochromes lack the PAS domain, while retaining photoconversion abilities, with some being able to photoconvert with just the GAF domain. Therefore, we investigated the ultrafast photoisomerization of the P More reactive, more efficient: The influence of the PHY domain on the photoconversion of a knotless phytochrome was investigated. The interaction of its “tongue” region with the GAF domain keeps the bilin chromophore and the binding pocket in a more reactive conformation. This results in accelerated reorganization of the protein environment necessary to accommodate the photoisomerization of the chromophore and, in effect, in an increased overall quantum efficiency. Z ⇄ E photoisomerization of the C15=C16 double bond and a subsequent rotation of the d‐ring of the embedded bilin chromophore .1, 2 TPhytochromes consist of chromophore‐binding GAF domains, PAS and PHY domains Figure .1, 21, 22r) and far‐red‐light (Pfr) absorbing, respectively.r→Pfr) photoconversion of these phytochromes proceeds with a lifetime of tens[Nostoc sp. PCC7120) is strongly distributed, and we assigned this behavior to conformational changes in the bilin‐binding pocket that control the photoisomerization of the chromophore.r→Pfr) photoisomerization dynamics of the complete knotless phytochrome All2699g1g2 .[Interestingly, single GAF domains of cyanobacteriochromes (CBCRs)e of tens19, 20r form of g1g2 has an absorption maximum at 637 nm, which is similar to that in g1,[fr form is significantly red shifted (by 73 nm) to 710 nm  spectra of both proteins (g1g2 vs. g1) have similar shape and undergo a sign inversion of the Q‐band CD signal upon Pg Figure .[8, 3The signs of these signals indicate the orientation of the peripheral rings A and D with respect to the co‐plane of rings B and C.8, 36 The role of the protein environment on the ultrafast photoisomerization dynamics of the PCB chromophore was investigated by femtosecond transient absorption (TA) measurements on g1g2 as compared with the single‐domain g1.r  with positive (420–575 nm) and negative (600–740 nm) amplitudes can be attributed to the simultaneous decay of these signals, and thus to the decay of PCompared with the positive‐amplitude distribution representing the decay of the ESA signal, the negative‐amplitude distribution, especially at 675 nm, appears stretched in lifetime. This can be explained by an overlaid additional negative‐amplitude distribution describing the rise of the primary photoproduct Lumi‐R commencing with the ES decay. On the scale longer than 1 ns, the negative and positive‐amplitude lifetime distributions correspond to the non‐decaying GSB and Lumi‐R signals.r* decay kinetics in the single‐domain g1 is described by broad and structureless lifetime distributions  of the first three states Figure  clearly r* decay kinetics. However, their rapid interconversion denotes that they are separated by low energetic barriers, and thus cannot explain the large excited state barrier that determines the relatively slow Pr* decay kinetics .r* decay. Therefore, our results point to a more dynamic picture of the kinetics in knotless phytochromes. In contrast, distinct ES decay components, including such on the sub‐50 ps timescale, have been reported for other phytochromes  and CBCRs, and were discussed in the framework of static ground state heterogeneity of the Pr form.[Recently, it was reported that in g1 and in g1g2 there exists a broad distribution of ground state subconformations that rapidly interconvert in solution.32, 48The comparison of the ultrafast dynamics of the g1g2 construct with the dynamics of the sole GAF domain g1r form  domain and categorically demonstrates the critical role of the protein environment on the photoisomerization step of the PCB chromophore.The similarity in the g1g2 and g1 early kinetics is in agreement with the lack of an immediate effect of the PHY domain on the steady state properties of the Pr* relaxation to proceed. This model explains the distributed character of the Pr* decay kinetics as it is imposed by the conformational dynamics of the protein. Eventually, Pr* decays (∼130 ps) yielding the primary photoproduct Lumi‐R.Based on the analysis presented above, we propose the following molecular picture for the photoisomerization dynamics of g1g2 Figure . After er* decay kinetics .Previously, we demonstrated that the ring D rotation of the PCB chromophore in g1 is hindered by a nearby Tyr residue (Tyr142).s Figure  and a mor* photoisomerization dynamics in knotless phytochromes. From an evolutionary perspective, the PHY domain “tongue” represents a development in phytochromes that tunes the photoreception efficiency. This is a key design principle for the development of optimized photoreceptors for biotechnological applications.Our work provides direct evidence for the essential role of the protein environment in the control of the photoisomerization kinetics of the PCB chromophore and outlines a detailed mechanistic picture of the PThe authors declare no conflict of interest.As a service to our authors and readers, this journal provides supporting information supplied by the authors. Such materials are peer reviewed and may be re‐organized for online delivery, but are not copy‐edited or typeset. Technical support issues arising from supporting information (other than missing files) should be addressed to the authors.SupplementaryClick here for additional data file.


Bearings are ubiquitous in rotating machinery and bearings in good working conditions are essential for the availability and safety of the machine. Various intelligent fault diagnosis models have been widely studied aiming to prevent system failures. These data-driven fault diagnosis models work well when training data and testing data are from the same distribution, which is not easy to sustain in industry since the working environment of rotating machinery is often subject to change. Recently, the domain adaptation methods for fault diagnosis between different working conditions have been extensively researched, which fully utilize the labeled data from the same machine under different working conditions to address this domain shift diploma. However, for a target machine with seldom occurred faulty data under any working conditions, the domain adaptation approaches between working conditions are not applicable. Hence, the cross-machine fault diagnosis tasks are recently proposed to utilize the labeled data from related but not identical machines. The larger domain shift between machines makes the cross-machine fault diagnosis a more challenging task. The large domain shift may cause the well-trained model on source domain deteriorates on target domain, and the ambiguous samples near the decision boundary are prone to be misclassified. In addition, the sparse faulty samples in target domain make a class-imbalanced scenario. To address the two issues, in this paper we propose a semi-supervised adversarial domain adaptation approach for cross-machine fault diagnosis which incorporates the virtual adversarial training and batch nuclear-norm maximization to make the fault diagnosis robust and discriminative. Experiments of transferring between three bearing datasets show that the proposed method is able to effectively learn a discriminative model given only a labeled faulty sample of each class in target domain. The research provides a feasible approach for knowledge transfer in fault diagnosis scenarios. Bearings are ubiquitous in rotating machinery and the failure of bearings would increase the downtime and operating cost. Fault diagnosis methods are of great importance to prevent accidents and reduce maintenance cost. These fault diagnosis methods could generally be divided into two categories: model-based methods and data-driven methods. The model-based diagnosis models usually develop dynamic models of the machinery through analytical approaches and finite element analysis. Gupta  introducIn real industrial applications, the working conditions of rotating machines, such as the rotation speeds and loading conditions, are subject to change during operations. The fault diagnosis model trained on one working condition could not generalize well to another working condition while testing, due to the distribution of test data are different from training data. It would be expensive and impractical to collect sufficient faulty data in industrial environments and retrain the models, especially for safety-critical systems. The existence of domain shift and the lack of bearing damages greatly hinder the application of fault diagnosis methods in industrial environments. Thus, the domain adaptation has been proposed to leverage annotated data from one domain, denoted as source domain, to learn the knowledge for another domain, denoted as target domain, with little or even no annotated data. Numerous studies have shown the benefits of knowledge generalization learned from different working conditions of the same machine. The domain discrepancy of feature representations was explicitly reduced by minimizing maximum mean discrepancy (MMD) ,6, multiHowever, when the target machine has little or even no annotated faulty data under any working condition, the domain adaptation methods between different working conditions are not applicable. Nevertheless, the annotated faulty data may be available in other machines, such as the open datasets or machines from laboratories. It is natural to explore generalizing fault diagnosis models from these different but related machines. Compared with diagnosis tasks between working conditions, the cross-machine diagnosis tasks are basically more challenging. The differences between two domains in mechanical structure of bearings, fault depth, and sampling frequencies lead to larger domain gap than traditional cross-domain fault diagnosis tasks. Less studies on cross-machine fault diagnosis have been carried. In  Yang et When larger domain gap exists and labeled target samples are insufficient, however, learning a domain-invariant model does not necessarily guarantee a target discriminative model. The well-trained model on source domain may deteriorate on target domain due to the large data density of target domain near the decision boundary, and the ambiguous target samples near the decision boundary are prone to be misclassified. To move decision boundaries into lower density regions of target domain, a variety of target discriminative methods have been explored. One approach is conditional-entropy minimization  which inAnother issue of cross-domain fault diagnosis is the class imbalance problem where the majority of target samples are normal samples and only limited number of faulty samples are available. When class imbalance exists, samples are prone to be classified as the majority category due to higher prior probability the majority category has. To handle the class imbalance, data-level methods, such as oversampling  or down-To cope with the reduction of discriminative and diversity of cross-machine fault diagnosis model, in this paper we employ the semi-supervised domain adaptation approach to fully leverage the sufficient unlabeled data and insufficient labeled data in target domain to improve the model performance. To handle the larger domain shift in cross-machine fault diagnosis tasks, an adversarial domain adaptation method utilizing nuclear norm maximization and variational adversarial training is proposed to achieve prediction discriminability as well as diversity. We also investigate the class imbalance scenario commonly exists in fault diagnosis tasks. Through this approach, only seldom faulty target data is needed to train an efficient fault diagnosis model.(1)An intelligent cross-machine fault diagnosis model is proposed. Fully utilizing the labeled data in source domain and unlabeled data in target domain, this model could perform fault diagnosis task effectively while only limited labeled target data is available.(2)We proved that the batch norm maximization is effective to improve the discriminability decline and diversity decline, which are both caused by the large domain shift.(3)Experiments between three open dataset of bearing faults are carried to verify the effective of the proposed method, especially under the situation that only 1 or 5 labeled samples from each category are available for target domain.The contributions of this work are:The remainder of this paper is organized as follows. The background of target discriminative domain adaptation approaches and batch nuclear-norm maximization are discussed in Domain shift often occurs when the models trained on one domain, denoted as source domain, and applied the models to another domain, denoted as target domain, which is obstacle to model generalization. Domain adaptation methods aim to align the feature distributions of two domains. Domain-invariant feature representations are usually learned in the shared feature space by minimizing the distribution discrepancy between domains. Tzeng et al.  exploresInspired by generative adversarial network, many achievements of domain adaptation have been achieved by adversarial training. A domain discriminator is trained to tell which domain the features come from, while the feature extractor is trained to confuse it. The domain discriminator and the feature extractor are trained adversarially until the target features are indistinguishable from the source features. In  Ganin etHowever, the annotated data for a target machine is not always available in industrial applications. The faults may have barely occurred if the target machine is new and works primarily under healthy conditions, so it is difficult to train an accurate fault diagnosis model independently. In this scenario, cross-machine diagnosis tasks are motivated where the source domain is from different but related machines. Compared with the cross-working condition diagnosis, the cross-machine diagnosis is less explored. In a deep convolutional transfer learning network (DCTLN) proposed by Guo et al. , domain Compared with conventional cross-domain fault diagnosis tasks which transfer from different working conditions of same machine, the cross-machine fault diagnosis tasks may face different mechanical structures, different fault characteristics, and different sampling frequencies between domains. The larger domain gap makes the cross-machine diagnosis tasks commonly more challenging.The existing cross-machine fault diagnosis methods focus on learning domain invariant feature distributions between source and target domain. However, learning a domain-invariant model does not necessarily guarantee a target discriminative model, especially when larger domain discrepancy exists and labeled target samples are insufficient. The well-trained models on source domain may deteriorate on target domain due to the large data density near the decision boundary, which violates the cluster assumption of semi-supervised learning. This could lead to annoying ambiguous predictions on target domain.To move decision boundaries into lower density regions and improve the model performance on target domain, a variety of target discriminative methods have been explored. These methods could be roughly categorized into two categories, namely discrepancy-based methods and conditional entropy-based methods. Saito et al.  proposedConditional-entropy minimization  is also As pointed by , howeverAnother issue of entropy minimization, as pointed out by ,41, is tAn effective and easy-to-implement method to simultaneously improve the target discriminability and diversity of model output is the batch nuclear-norm maximization (BNM). In  Cui et aWe propose to use Frobenius-norm (F-norm) of prediction output as indication of target discriminability. F-norm has strict opposite monotonicity with the entropy roved in . IncreasWe measure the feature diversity of domain adaptation by nuclear norm of the features. Assume that two randomly selected vectors, As proven in , the relTo jointly increase the discriminability and diversity of the prediction outputs, we incorporate the batch nuclear-norm maximization (BNM) regularization into model training. Within each batch samples In this section, we introduce a semi-supervised domain adaptation method for the cross-machine fault diagnosis task. To align the distributions between domains and perform fault diagnosis on target machine with very limited annotated data, the fault diagnosis model learned from related but different machines are generalized. The framework of the proposed method is illustrated in E, the label classifier C, and the domain discriminator D. The feature extractor E learns a function C learns a function D learns a function E and domain discriminator D where the domain discriminator tries to distinguish which domain the samples come from while the feature extractor tries to fool it.The adversarial domain adaptation achieves distribution alignment through adversarial training between feature extractor and domain discriminator. Specifically, the adversarial training involves three components, namely the feature extractor E and classifier C on the labeled samples and their corresponding labels in a supervised way, by minimizing the cross entropy loss with During training, given access to labeled samples in the source domain and limited labeled samples in the target domain, we first train the feature extractor We now train the feature extractor and domain discriminator adversarially. The domain discriminator takes the feature representation and domain label as input, where the samples from source domain is donated as label 0, and the samples from target domain is donated as label 1. The output is the predicted domain label, To perform domain alignment, the gradient reversal layer proposed in  is emploTo jointly increase the discriminability and diversity of the prediction outputs, we incorporate the batch nuclear-norm maximization (BNM) regularization into model training, the classification output could be denoted as To improve the stability of adversarial training and improve the prediction performance, we further impose the local Lipschitz constraint by virtual adversarial training (VAT). VAT perturbs the target data in the most sensitive direction of the model at the input level and minimize the adversarial loss so as to improve the robust of model. The VAT minimization objective is:The total loss for the model is defined as:Algorithm 1: Details of the proposed methodRequire: source data 1.Initialize feature extractor, domain critic, classifier with random weights 2.for3.Sample labeled minibatch 4.calculate the cross-entropy loss: 5.calculate the BNM loss:6.calculate the VAT loss on target samples: 7.calculate the discriminator loss: 8.calculate the overall objective: 9.untilThe algorithm details of the proposed method are presented in Algorithm 1.The first bearing dataset in use is provided by Case Western Reserve University (CWRU) Bearing Data Center . The vibThe IMS bearing data  providedThe third bearing data comes from the centrifugal fan system for rolling bearing fault diagnosis testbed of JiangNan University . The beaThe three testbeds are shown in To prepare the training and testing data, we use a sliding window of size 1024 to scan the raw signals and generate the data samples, hence each sample is composed of 1024 sequential points. Each category has 2000 samples, and description of the datasets in use are shown in The network architecture of proposed method consists of three 1-D convolutional layers, following the rectified linear units (ReLU) activation function and the batch normalization layer. The representation is then flattened and passed to label classifier and domain discriminator. The detailed network architecture is shown in To validate the performance of the proposed method, we compare our method with the following methods. To be fair, the neural network used in our method and the compared methods are kept the same.CNN: The model trained on labeled source domain data is used to classify the target samples without domain adaptation.Domain adversarial neural network (DANN) proposed by Ganin et al. . FeatureDCTLN proposed by Guo et al. . AdversaVADA proposed by Shu et al. . VADA inDANN + Entropy Minimization (EntMin). The discriminability of model is improved by entropy minimization on the basis of adversarial training.DANN + BNM (BNM). The discriminability of model is further improved by batch nuclear-norm maximization on the basis of adversarial training.The reported experimental results are averaged by 5 trials to reduce the effect of randomness, and the mean values are provided. All the experiments are implemented using Pytorch and running on Nvidia GTX 2060 GPU.We evaluate the cross-domain fault diagnosis methods when different numbers of annotated target data are available. The performances on different transfer tasks and different number of labeled target samples are listed. To better compare the performances of different methods, we drew the performances of different methods on different tasks when number of target samples is equal to 1 and 5, respectively. When number of labeled target samples is 1, as shown in We draw the representations of DANN and proposed method via t-SNE for comparison in To further explore the cross-domain diagnosis tasks under class-imbalanced scenarios, we explicitly reduce the unlabeled faulty samples in target domain. Three imbalanced scenarios are composed with different imbalance rate, as shown in From the results we could find that, the imbalanced scenarios have negligible effect on tasks where CWRU and IMS are target domains. For tasks where JNU is the target domain, the proposed method has obvious improvement over compared methods.Environmental noise commonly exists in industrial environment, and the vibration signal will be inevitably affected by the noise, which may deteriorate the performance of the fault diagnosis methods. Since the target machine may suffer a different level of noise disturbance, in this section, we evaluate the fault diagnosis performance under environmental noise. The testing vibration signal is added with the additive white Gaussian noise of different signal-to-noise ratio (SNR) which is defined as:We test the different diagnosis models under noise level from −4 dB to 4 dB. The result of task ‘DE -> IMS’ is shown in It could be found that the diagnosis accuracies decrease as the signal-to-noise ratio decreases. Among these fault diagnosis models, the proposed method has better performance on both two tasks. This can be explained that the ambiguous samples in our model are better pushed away from the decision boundaries.In this paper, we leverage semi-supervised domain adaptation method to address the cross-machine fault diagnosis problem. Cross-machine fault diagnosis is a promising method to address the dilemma of labeled data insufficiency in data-driven fault diagnosis tasks, especially when labeled data from different working conditions of same machine are unavailable. While it provides the opportunity to generalize the fault diagnosis knowledge from different machines, the cross-machine fault diagnosis is generally more challenging, since the large domain gap between machines make the ambiguous samples near decision boundary are prone to be misclassified, and the data imbalance scenario in target domain makes the faulty samples are prone to be misclassified as normal healthy category. In our method, feature alignment is achieved by adversarial domain adaptation, then the ambiguous samples near decision boundary are further reduced by batch nuclear-norm maximization and virtual adversarial training. Experiments on three different bearing diagnosis scenarios verify the efficacy of our proposed approach when only one labeled target samples each class are available. The class imbalance scenarios also are investigated.Although the proposed method is effective for cross-machine fault diagnosis, some limitations exist when applying this method. The semi-supervised method assumes there are a few labeled samples in target machine. In our method, adding one labeled sample to each class of target dataset will dramatically increase the accuracy of model prediction. However, in real applications it may be difficult to obtain these labeled data since it is sometimes expensive or infeasible to perform the experiment. In this case, generating labeled data using simulation or digital twin may be a feasible method. Through the virtual experiments, the simulation data could be easily collected, and the annotation information is also available, but with the cost of larger domain gap between simulation data and sensory data. In the future, we plan to explorer the domain adaptation tasks between simulation signals and real signals.


This study aimed to discover whether using maltogenic amylase (MAse) to modify starch in germinated brown rice flour may enhance slow digestion starch and release more bioactive compounds (BCs) content. To achieve this aim, the starch was modified with four levels of MAse  for 1 hr at pH 5 and then spray‐dried to make modified flour. The biochemical impacts of the products were then accessed in normal and type 2 diabetic mice for 4 weeks. The result showed that when the starch was modified by MAse 266 U/g, a significant reduction of rapidly digested starch to 22.35% from 61.56%, an increase in slowly digested starch to 33.09% while resistant starch as 2.92% corresponding to the increase of γ‐amino butyric acid to 528.1 ± 44.1 mg/L and 120.6 ± 10.9 mg/L of ferulic acid. The extract from modified flour showed very strong cytotoxic activity against HepG2 cell (>80% inhibition). The result in vivo showed that the type‐2 diabetic mice fed with this modified product could better improve the stability of the glycemic index. Also, atherosclerotic plaque assessment further supports these findings. The results indicated that BCs released considerably couple with the changes in starch properties caused by MAse enhanced the effectiveness of this product to diabetes as well as positive effect on cytotoxic activity against HepG2 cell. p < .05). The uppercase letters  in superscript indicated a statistically significant difference between the time frames of each treatment by row (p < .05). NS: normal mice fed nonmodified GBR flour, MA: normal mice fed GBR flour modified, T2D‐NS: type 2 diabetic mice fed nonmodified GBR flour, T2D‐MA: type 2 diabetic mice fed GBR flour modified, T2D‐NS‐M: type 2 diabetic mice fed combination nonmodified GBR flour with metformin and T2D‐MA‐M: type 2 diabetic mice fed combination GBR flour modified with metforminThe lowercase letters  in superscript indicated a statistically significant difference between treatments in each time by column ( All the solvents used in the experiments, 4‐dimethyl aminoazobenzene‐4‐sulfonyl chloride, 5‐sulfosalicylic acid dihydrate are analytical reagents, or HPLC grade and were purchased from Merck, Fisher and J.T.Baker‐Mallinckrodt Baker Chemical. At once, glucoseamylase , alpha‐amylase  of Novozymes (Novozymes A/S). Sulforhodamine B (SRB) from Sigma and other chemicals with purity level for the study were provided by CEMACO joint‐stock company.The GABA, FA, GORZ, enzymes such as MAse  and 0.6% glutamic acid and incubated in an incubator (SANYO MCO‐5AC Incubator) which had a maximum yield of 0.5 kg/batch at a temperature of 37°C with 24 hr incubation period. Thereafter, the GBR was dried at 60°C for 4 hr and milled into flour, packaged and stored at −18°C for use in the experiments.The paddy rice IR 50,404 variety was received in Cuu Long Delta Rice Research Institute in Can Tho city, Vietnam and stored at the temperature of 20 ± 2°C) and dehusked  to brown rice; then, the rice seeds were soaked in a buffer solution with pH 4 , was added to achieve a mixture of 20% substrate, and then stirred well. The sample flasks were heated to 80–85⁰C in a boiling water tank with 90–95⁰C temperature to gelatinize the starch. After 20 min, the samples were cooled down and incubated in the water bath set at 60⁰C. The addition of enzymes to the corresponding labeled Erlen with 0 U, 133 U, 266 U, and 399 U MAse/g dry weight of GBR‐based flour, respectively, was performed after having a thermal balance between inside and outside. The incubation times were 0 and 1 hr. Subsequently, the samples were filtered through a curtain cloth to get the soluble components and to remove residue before analysis, then stored at −18⁰C for further analysis of BCs and digestibility indexes of starch. The remaining samples were dried by Spray Dryers  to get MGBRF and then stored at −21°C in the freezer, Sanaky, Vietnam.2.4The cytotoxic activity against HepG2 cell was done following the method of Skekan et al. . Briefly2.53 and 100 μl of 4 mM 4‐dimethylaminoazobenzene‐4‐sulfonyl chloride (was dissolved with acetonitrile solvent) to Eppendorf, then vortexed. The mixture was heated at 70°C for 20 min to derivatization effect, after that, it was cooled down and added in 0.5 ml of absolute ethanol on ice and centrifuged at 16,000 g and 4°C for 5 min. Finally, the supernatants were analyzed on HPLC (Shimadzu), with Ascentis® C18 HPLC column, 25 cm × 4.6 mm, 5 µm (Supelco). The HPLC was equipped with a UV‐Vis detector  set at 465 nm wavelength. The mobile phases were 25 mM ammonium acetate buffer containing 0.1% acetic acid and acetonitrile (26:74) adjusted at the flow rate of 1 ml/min with 10 µl of injection volume and temperature column oven at 55°C. Pure GABA was used as a calibration standard.The GABA was separated and analyzed by the method of Banchuen et al.  with sli2.6Total FA content was determined using high‐performance liquid chromatography as described previously , slowly digestible starch (SDS) and resistant starch (RS) based on the rate and extent of its digestion in the in vitro Englyst assay . The mobile phase conditions of UPLC for GABA and FA were A: 100% methanol and B: water. The injection volume was 10 μl. The retention time was 5 min. The column was ACQUITY UPLC CSH C18 1.7 μm, 2.1 × 50 mm from Waters. For GORZ analysis, the column was Luna 5 μm NH2.11SD 06, Labplant UK Ltd, UK), with peristaltic pump: 485 ml/h and stored at −18⁰C in PA vacuum packaging. Each sample (450 mg/kg body weight) was dissolved into 5 ml of boiling water to feed the mice. The nutritional components of samples in this experiment are shown in Table 42 Swiss strain and diabetic mice  with ages 6–8 week, 25–30 g ideal weight used for the experiments were bought from the Pasteur Institute of Nha Trang, Vietnam. The mice were housed in an environmentally controlled room  and fed ad libitum with chow diet  for healthy maintenance. Optimal samples from the experiments, investigating changes in BCs and in vivo results, were spray‐dried at 160°C (LabPlant‐2.12Test samples included glucose (Glu), sucrose (Suc), modified (MA‐S) and nonmodified (NS) GBR base flour. Glucose tolerance (GT) test in mice was carried out using the method of Dura et al. . Four mi2.13M (type 2 diabetes group fed with nonmodified spray‐dried diet and metformin), and T2D‐MA‐M (type 2 diabetes group fed with modified diet and metformin). The mice in the group using metformin were given hypoglycemic agents  and pumped 0.2 ml directly into the esophagus after 2 hr using sample test. Feeding lasted for four weeks and the result was recorded every week. The samples and experiments were administered using a 4 × 4 Latin square design. Recorded results and blood glucose units were calculated according to Joslin Diabetes Center, US website .There were three animals per cage, and they were divided into six groups: NS , MA (noninduced type 2 diabetes fed with modified diet), T2D‐NS (type 2 diabetes group fed with nonmodified), T2D‐MA (type 2 diabetes group fed with modified diet), T2D‐NS‐2.14Total cholesterol (TC), triglycerides (TG), High‐Density Lipoprotein Cholesterol (HDL‐C), low‐density lipoprotein cholesterol (LDL‐C), alanine aminotransferase (ALT), aspartate transaminase (AST) in blood, malondialdehyde (MDA), glutathione reductase (GSH) in liver homogenates were determined with enzymatic methods using corresponding Ams Ellipse Automated Analyzer .2.15p ≤ .05) was used to determine the significant difference between treatments.Results are expressed as mean ± standard deviation and all experiments were conducted in triplicates. Data were calculated on average and the standard deviation by Excel 2010 software. Data were statistically analyzed for ANOVA using the Statgraphics Centurion XV software . LSD test at a 5% significance level . The components rose together by advancing MAse activity at 266 U MAse/g dry weight of GBR‐based flour, both of them had the highest level with 528.1 mg/L of GABA and 120.6 mg/L of FA but slightly decreased at the level of 399 U MAse/g dry weight of GBR‐based flour. So, when using the enzyme MAse to modify the GBR base flour showed that there was a positive impact in its ability to release functional compounds. In this study, the GBR‐based flour was used as material for modification by MASe. After enzyme reaction, the mixture was filtrated by a curtain cloth to get the soluble components as well as discard the water insoluble component or solid. So, mostly nonmodified and less‐modified starches as insoluble components were kept in this insoluble component part while released protein as well as bioactive compounds into soluble part. It was then used for spray drying to make MGBRF. As a result, there were an increase of protein and bioactive compounds in the MGBRF comparing to nonmodified spray‐dried flour as shown in Table Streptomyces spp. This enzyme combination also had an effect on FA production from other biomasses, such as raw rice bran, wheat bran, and corncob. Moreover, an enzymatic hydrolysis process for FA extraction from wheat bran was optimized gradually by enzymatic pretreatment with alcalase and termamyl, and a final enzymatic hydrolysis with pentopan and feruloyl esterase extraction yield higher than untreated condition , slowly digestible starch (SDS) and resistant starch (RS) based on the rate and extent of its digestion in the in vitro Englyst assay . And type 2 diabetic mice had a GI level significantly higher than normal mice (p < .05). The diet of modified GBR flour had the lowest level of GI in both nondiabetic and diabetic groups than the control diet (nonmodified flour). Though GI parameters slightly reduced, there was no difference between control diet and two experimental diets (modified GBR flour and nonmodified GBR flour with metformin) in diabetic mice (p < .05). In the diabetic mice population, on the other hand, when those fed with the modified GBR flour combined with metformin was compared to those fed with the existing diets showed a statistically significant decrease in GI level of 14.37 in the fourth week (p < .05). Meanwhile, the GI level of diets including nonmodified GBR flour, modified GBR flour and nonmodified GBR flour with metformin were 26.37, 22.1 and 20.1, respectively. Besides, the diabetic mice fed with modified GBR flour showed that blood glucose was maintained at the initial level. GI is an indicator of postprandial hyperglycemia of carbohydrate food and reflects the rate at which glucose is absorbed from the gastrointestinal tract into the bloodstream and cellular tissues .The data in Table Roohinejad et al.  also dem3.7Table p < .05). According to Esa et al. (M (p < .05). For the diabetic group, the levels of MDA in groups fed with modified GBR flour decreased significantly compared to nonmodified GBR flour. However, it was still greatly higher than normal mice. Moreover, this parameter occupied the lowest level in normal mice fed with nonmodified GBR flour. Regarding GSH content, it had a trend contrast to MDA content with similarity progression for all groups. It was noticed that GSH content in normal mice was significantly higher than diabetic mice. And this content was improved when fed with modified GBR flour. As a role, using metformin proved more effective and enhanced if combined with diet intake of modified GBR flour. Mohd‐Esa et al. (p < .05) in the group supplemented with GBR at week 10 compared to week 5. On the other hand, GORZ possesses a cholesterol‐lowering property when fed to humans and animals through decreasing cholesterol absorption and enhancing fecal sterol excretion , Vietnam dated in 15th July, 2019.