When an elastic material  is horizontally stretched (or compressed), the material is compressed (or extended) vertically – so-called the Poisson effect. In the different case of the Poisson effect, when an elastic material  is vertically squashed, the material is horizontally extended. In both cases, the visual system receives image deformations involving horizontal expansion and vertical compression. How does the brain disentangle the two cases and accurately distinguish stretching from squashing events? Manipulating the relative magnitude of the deformation of a square between horizontal and vertical dimensions in the two-dimensional stimuli, we asked observers to judge the force direction in the stimuli. Specifically, the participants reported whether the square was stretched or squashed. In general, the participant’s judgment was dependent on the relative deformation magnitude. We also checked the anisotropic effect of deformation direction  and found that the participant’s judgment was strongly biased toward horizontal stretching. We also observed that the asymmetric deformation pattern, which indicated the specific context of force direction, was also a strong cue to the force direction judgment. We suggest that the brain judges the force direction in the Poisson effect on the basis of assumptions about the relationship between image deformation and force direction, in addition to the relative image deformation magnitudes between horizontal and vertical dimensions. Human observers can easily distinguish rigid and non-rigid materials. Previous vision science literature has shown that spatial motion patterns are diagnostic features for an observer to perceive non-rigid materials. For example, This study focuses on the simple but important physical behavior of materials, the so-called Poisson effect. When an external force is applied to an elastic material , the matdx is the change in length due to longitudinal strain and dy is the change in length due to transverse strain . On the other hand, if the material is thin, the secondary horizontal axis can be ignored. From this point of view, two-dimensional stimuli including deformation along two spatial axes were used as stimuli in this study. As described above, the horizontal stretching of an elastic material causes the vertical compression in one situation of the Poisson effect. On the other hand, in the other situation of the effect, the vertical squashing of an elastic material causes a horizontal expansion. Importantly, in both situations, the Poisson effect involves an image deformation with horizontally outward and vertically inward vectors  surfaces of an elastic material. However, in the real-world situation, the material to which an external force is vertically applied is typically located on a plane . Under the scenario, the material deformation is expected to be anisotropic . SpecifiIn total, 822 people participated in the experiment. Their mean age was 40.1 (SD11.33). They were recruited online by a crowdsourcing research agent in Japan and paid for their participation. Only the people who could participate in the experiment using their personal computers were recruited. They were unaware of the specific purpose of the experiment. Ethical approval for this study was obtained from the ethics committee at Nippon Telegraph and Telephone Corporation . The experiments were conducted according to principles that have their origin in the Helsinki Declaration. Written informed consent was obtained from all observers in this study.Stimuli were movies, each of which consisted of a fixation followed by the animation of deforming a two-dimensional grayscale random noise square see . The stiThe noise square subtended 256 × 256 pixels. The size of each noise cell was 16 × 16 pixels, and hence, the noise square consisted of 16 × 16 noise cells. The intensity of each noise cell was randomly chosen from the uniform distribution ranging between 0 and 255.dmax) was first determined. Second, the deformation magnitude of the square on the nth frame of the animation (dn) was determined by the following formula,The noise square deformed in the horizontal as well as vertical directions. To deform the square, we used a conventional pixel warp method with a linear interpolation of pixel values . Figure n ranged from 1 to 30. As shown in dmax was determined separately for horizontal and vertical dimensions. The conditions of 20 pixels dmax in the horizontal direction and 20 pixels dmax in the vertical direction were duplicated.wherein We manipulated the direction of deformation vectors between horizontal and vertical dimensions. In one condition, that is, the horizontally-outward-vertically-inward (HOVI) condition, the deformation vectors were outward and inward for horizontal and vertical dimensions, respectively. In the other condition, that is, the horizontally-inward-vertically-outward (HIVO) condition, the deformation vectors were inward and outward for horizontal and vertical dimensions, respectively. Apart from the direction of deformation vectors, we also examined the effect of symmetricity of motion vectors between horizontal and vertical deformations. In the Symmetry condition, both the vectors of horizontal  and vertical  deformations were inward and outward, respectively. In the Asymmetry condition, however, the vectors of horizontal and vertical deformations were asymmetric, as shown in dmax conditions  ratio of deformation magnitudes, as shown in F = 64.582, p < 0.0001, ηp2 = 0.08]. The main effect of the vector symmetricity was not significant . Interaction between the two factors was significant . The simple main effect of the vector direction showed that the threshold in the HOVI condition was significantly lower than the one in the HIVO condition for the Symmetry condition  and the Asymmetry condition . The simple main effect of the vector symmetricity showed that the threshold in the Symmetry condition was significantly lower than the one in the Asymmetry condition for the HOVI condition . On the other hand, there was no significant difference in the threshold between the Symmetry and Asymmetry conditions in the HIVO condition .The SS ratio differs from the Poisson’s ratio in several ways. First, if the original material shape before deformation is not square, the SS ratio takes on a different value than the Poisson’s ratio. Second, if the material is assumed to be squashing, the SS ratio is equal to the Poisson’s ratio, but if the material is assumed to be stretching, the Poisson’s ratio is the inverse of the SS ratio in our stimulus setting. Third, as stated in the “Introduction,” the Poisson’s ratio of natural materials does not exceed 0.5. Because the SS ratio is not subject to the laws of physics, it exceeds 0.5. We chose to use the SS ratio instead of the Poisson’s ratio in this study because we would like to examine how the relative magnitude of image deformation between the horizontal and vertical dimensions contributed to the judgment of the stretching-squashing events in the Poisson effect. When the SS ratio was 1, the judgment of stretching/squashing events would be ambiguous if the observers relied their judgments on the relative magnitude of image deformation. We also examined how the judgment was biased by the vector symmetricity and vector direction. As shown in t-test to check whether the threshold deviated from 1. When the SS ratio was 1, the magnitude of deformation was equivalent between squashing and stretching. If the visual system purely used the relative magnitude of image deformation between the horizontal and vertical dimensions, the threshold should not deviate from 1. In the HOVI Symmetry condition, the threshold was significantly lower than 1 . In the HOVI Asymmetry condition, the threshold was significantly lower than 1 . In the HIVO Symmetry condition, the threshold was not significantly deviated from 1 . In the HIVO Asymmetry condition, the threshold was not significantly deviated from 1 .Additionally, we conducted a one-sample We examined how the human observers judged the force direction in the Poisson effect.Consistent with our prediction, the proportion of reports for stretching relied strongly on the SS ratio . The resOn the other hand, we observed an interesting anisotropy in the force direction judgment. In both the HOVI Symmetry and HOVI Asymmetry conditions, the threshold significantly deviated from 1. The results indicate that the judgment was biased toward stretching when horizontally outward deformation was given to the noise square. It is unclear why this sort of anisotropy occurred in the judgment of the Poisson effect. In the HOVI condition stimuli, the vertical deformation vector was inward while the horizontal deformation vector was outward. In general, inward and outward motion vectors are involved with contraction and expansion motion. Previous studies investigating complex motion perception have been shown that the detection and interpretation of expansion motion were prioritized over those of contraction motion . If the The vector asymmetry also contributed to the determination of the judgment bias. The threshold in the HOVI Symmetry condition was significantly lower than the one in the HOVI Asymmetry condition. The results are consistent with our hypothesis that the asymmetric vector pattern, which was involved with the situation, as shown in In this on-line experiment, neither the observation distance nor the screen size was controlled. Therefore, it is an open question whether the stimulus size can affect the stretching/squashing of the judgment in the Poisson effect. Here, we would like to discuss some perspectives to support that manipulation of viewing distance or stimulus magnitude is not important in subjectively determining squash-stretch events. First, the variation of viewing distance and/or stimulus sizes equally influences the deformation magnitude of the horizontal and vertical dimensions in a retinal image, and this will not affect the ratio between them. Second, in the real world, the Poisson effect can occur for the objects of various sizes. When the brain learns the statistical properties of the Poisson effect, learning will not depend on the size of objects. This is because the Poisson effect occurs at various scales, so the learning is possibly independent of object sizes. Consistent with this idea, our results showed that the reported proportion of stretching/squashing judgment was stable across the observers . Although the physical behavior of the Poisson effect is often described by the two-dimensional pattern, the Poisson effect of a three-dimensional material causes a change in visual signals useful to a human observer who judges the material properties. For example, deformation of the three-dimensional surface often results in a local change in the three-dimensional waveform, which results in a change in the specular and/or refractive pattern. Such changes in optical information are powerful clues to material perception . The texAs speculation, we suggest that there may be an internalized assumption or “a prior” about the relationship between the force direction and resultant image deformation. For example, the brain may have an assumption that stretching is related to horizontal vectors. For example, the stretching behavior, such as stretching the cloth, is mainly done with both hands on the horizontal axis. The assumption may underlie the significant difference of threshold from 1 in the HOVI condition. The vector asymmetry was also effective in the HOVI condition only. This may also come from a different assumption that the vector asymmetry, as shown in The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.The studies involving human participants were reviewed and approved by NTT Communication Science Laboratories Research Ethics Committee. The patients/participants provided their written informed consent to participate in this study.TK designed and conceived this study, wrote the manuscript, and finalized the manuscript for submission.The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Multi-domain proteins (MDPs) show a variety of domain conformations under physiological conditions, regulating their functions through such conformational changes. One of the typical MDPs, ER-60 which is a protein folding enzyme, has a U-shape with four domains and is thought to have different domain conformations in solution depending on the redox state at the active centres of the edge domains. In this work, an aggregation-free small-angle X-ray scattering revealed that the structures of oxidized and reduced ER-60 in solution are different from each other and are also different from those in the crystal. Furthermore, structural modelling with coarse-grained molecular dynamics simulation indicated that the distance between the two edge domains of oxidized ER-60 is longer than that of reduced ER-60. In addition, one of the edge domains has a more flexible conformation than the other. MDPs can have several domain conformations and dynamically transit between them under physiological conditions, in solution. The domain conformation plays a role in the association with other proteins, nucleic acids, small molecules, etc., therefore determining a specific MDP function. Thus, elucidating the domain conformations and their dynamics in solution is important.Multi-domain proteins (MDPs) constitute one of the major categories of ubiquitous proteins, presumably accounting for over half of all human proteins13, which is a protein folding enzyme belonging to the protein disulphide isomerase family. ER-60 is also a typical MDP consisting of four thioredoxin-like domains, a, b, b′, and a′15. Two edge domains, a and a′, have catalytically active Cys–Gly–His–Cys (CGHC) motifs, which interconvert between oxidized (disulphide) and reduced (dithiol) forms. The oxidized form catalyses the oxidation reaction of dithiol on a substrate (an unfolded protein), whereas the reduced form catalyses the reduction or isomerization of disulphide bonds on a substrate. In this catalytic process, ER-60 must have appropriate domain conformations to perform the formation and rearrangement of disulphide bonds on the substrate16. Therefore, it is assumed that ER-60 can change its domain conformation according to the redox state of the CGHC motif to either oxidize or reduce the substrate.We have focused on ER-60, also called ERp57, GRP58, or PDIA318. As shown in Supplementary Fig. a and a′. This suggested that, under physiological conditions, namely in solution, the domains of ER-60 could also form a similar U-shape and associate with a substrate using the a and a′ domains, and that the conformation of the a and a′ domains could be changed according to the redox state of the CGHC motif. To confirm this, the structure of ER-60 in solution, especially the conformation of the four domains, and its dependence on the redox state of the CGHC motif, must be revealed.The atomic-scale structure of full-length human ER-60 has only been revealed in complex with tapasin in an X-ray crystallographic study21. Therefore, we carefully prepared highly purified oxidized and reduced ER-60, examined the quality of the samples, and conducted state-of-the-art SAXS methods combined with size-exclusion chromatography (SEC) (SEC-SAXS)22 and with analytical ultracentrifugation (AUC) (AUC-SAS)21. The integrative organization of these methods allowed us to achieve high-precision aggregation-free SAXS profiles.For this purpose, small-angle X-ray scattering (SAXS) is a powerful tool that can be used to observe the structure of biomacromolecules in solution. However, SAXS is sensitive to unfavourable contaminations and aggregates in solution, especially large molecules when even in small amountsb–b′ domains: one is the b–b′ domains of full-length ER-60 in the tapasin complex (3F8U) and the other is only b–b′ domains, ER-60 truncated the a and a′ domains23 (2H8L)  simulations to search for the possible structural space of ER-60. Compared to standard atomistic MD simulations, the CG-MD approach drastically accelerates structural space sampling25. The sampled models were screened with the aggregation-free SAXS profiles of the oxidized and reduced ER-60 forms.To visually understand domain conformation, we built three-dimensional structures of oxidized and reduced ER-60 in solution by computational modelling using the SAXS data. Prior to the computational modelling, we compared two crystal structures of the Finally, we succeeded in deriving two sets of structural models from simulation snapshots, which represented the solution structures of oxidized and reduced ER-60, respectively. From the structural models, we examined the differences in their domain conformations.2 between the observed SAXS profiles and that calculated using the crystal structure were 13.5 and 10.7 for oxidized and reduced ER-60, respectively. This means that the complex-free structures in solution are different from the crystal structure of the complexed protein.Figure Rg with Guinier analyses and the longest distance Dmax from the distance distribution functions are presented in Table Rg, the longer Dmax, and the smaller elution volume of the oxidized ER-60 compared to reduced ER-60 clearly indicated that oxidized ER-60 forms a more expanded structure than reduced ER-60 in solution.Next, we examined the structural differences between oxidized and reduced ER-60. Their Guinier plots and distance distribution functions are displayed in Fig. 2, there are no large differences between the three structures: the oxidized and reduced ER-60, and the crystal structures of the complexed protein. However, we have succeeded to find the small but certain differences with our careful sample preparation and high-precision SAXS measurements.In summary, in terms of χWe built structural and dynamical models for both forms, which reflect their differences. Considering the structural differences between two forms and the crystal form are not large, it is assumed that the solution structures for both forms could distribute homogeneously around the crystal structure. Therefore, we extensively sampled structures with various domain conformations by CG-MD. Subsequently, the structures that reproduced the experimental SAXS profiles for both forms were collected from the sampled structures.2 < 7.0 and 31.9 Å < Rg < 32.3 Å. The total number of ox-models was 230. The structural models reproducing the observed SAXS profile of reduced ER-60 (red-models) were extracted with two evaluation criteria: χ2 < 7.0 and 30.6 Å < Rg < 31.0 Å. The total number of red-models was 1572. Figure First, we took 800,000 snapshots with CG-MD  of the a and a′ domains. As shown in Fig. θa–b–b′ showed similar ranges distributed around 75º in both models, whereas θb–b′–a′ showed relatively different ranges, which were 110°–155° and 95°–150° for ox- and red-models, respectively. This suggested that the a domains of the ox- and red-models have almost the same position, while the a′ domains of the ox-models locate at a more open position than those of the red models. Moreover, as shown in Fig. φa–b–b′–a′ had a similar range distribution in both models, meaning that the degrees of twisting of the U-shape were almost the same. The effect of the more open conformation in the ox-models is clearly recognized in the Da–a′. As shown in Fig. Da–a′ of the ox-models was distributed over a longer range than that of the red models.Subsequently, we investigated the differences in domain conformation between the ox- and red-models. Three angles and one distance, which characterise domain conformation, were calculated for the ox- and red-models. As shown in Supplementary Fig. b and b′ domains of all models were superimposed, and then the positions of the COM of every domain were averaged over the ox- and red-models, respectively. The COM-positions were expressed with small spheres and the subscripts for a and a′ domains, o and r, corresponding to ox- and red-model(s), respectively. As shown in Fig. ao and ar are almost the same, but a′o is slightly shifted to be more open than a′r. The distance between averaged ao position and averaged ar position was 3.4 Å, whereas that between averaged a′o and averaged a′r was 4.7 Å. This also suggests that the structures of oxidized and reduced ER-60 mainly differ in the position of the a′ domain. The results are summarized in Table The structural images are shown in Fig. a′ domain. As shown in Fig. a′ domains in both the ox- and red-models were distributed keeping the distances Da–a′. This implied that the a′ domain rotates around the a domain, which results in the twisting of the U-shape because the a domain is almost fixed. This motion is shown in Fig. a′ domain shifts from θb–b′–a′ = 140° to θb–b′–a′ = 110°, meaning that the a′ domain rises up to the a domain, the dihedral angle changes from φa–b–b′–a′ = 0° to φa–b–b′–a′ = 60°, meaning that the flat U-shape is being twisted  correlated with the position shift of the a′ domain (Δθb–b′–a′ = 30°). As shown in Fig. θa–b–b′, θb–b′–a′, φa–b–b′–a′) were 80°, 140°, 0° and 70°, 110°, 60°, respectively. The difference in conformational flexibility between the a and a′ domains could be ascribed to the linkers between the b and b′ domains. In the red-models, same conformational relations such as triple angle relations  were observed from 80°, 140°, 0° to 70°, 110°, 60°. The difference from the ox-models is the shorter Da–a′ as described. These structural pictures are shown in Fig. θa–b–b′, θb–b′–a′, φa–b–b′–a′, or Da–a′  and cleavage (reduction) of disulphide bonds.We also compared the ox- and red-models with the crystal structure. In Fig. In summary, the combination of aggregation-free SAXS and CG-MD simulations is a powerful tool to analyse the structure of MDPs in solution, especially for domain conformation fluctuation.a and a′ domains are different . Interestingly, the a′ domain could fluctuate more than the a domain, may allow twisting of the U-shape. We speculate that the twisting resulting from the fluctuation of the a′ domain may allow ER-60 to capture a variety of substrates. To probe this will be the next research target.Aggregation-free SAXS data of oxidized and reduced ER-60 were acquired by integrating the following procedures: highly purified sample preparation, step-by-step sample quality checks, SEC-SAXS, and AUC-SAS. The SAXS data indicated that both solution structures are different from the crystal structure of the tapasin-ER-60 complex and that the structure of oxidized and reduced ER-60 differs; the former is slightly more expanded than the latter. To analyse the structures and fluctuations of their domain conformations in more detail, three-dimensional structures were constructed with CG-MD simulation, which allows searching for a large phase space within a relatively short time compared with the all-atom MD simulation. CG-MD analysis revealed that the U-shape in the crystallographic structure is held in solution, but the distances between the We developed several methods to obtain aggregation-free SAXS data. Figure 26. Briefly, E. coli BL21(DE3) cells  transformed with expression plasmids encoding mature human ER-60 (Ser25-Leu505) without signal peptide were precultured in LB broth at 37 °C for 5 h. The expression was induced in the presence of 0.4 mM isopropyl-β-d-thiogalactopyranoside at 25 °C for 48 h. Purification procedures of the recombinant ER-60 were carried out at 4 °C. E. coli cells suspended in buffer A  were disrupted by sonication and centrifuged at 10,000 × g at 4 °C. The expressed recombinant ER-60 was purified from the cell lysates using affinity column chromatography on AF heparin toyopearl 650 M  with buffer A at 4 °C. Recombinant ER-60 was eluted from the AF heparin column with 0.4 M KCl in buffer A and subsequently purified by ion-exchange column chromatography on Resource Q  with 20 mM Tris–HCl, pH 7.4 buffer containing 0.2 mM EDTA and 0.5 mM phenylmethylsulphonyl fluoride, followed by SEC on Superdex 200 Increase 10/300GL  with SEC buffer  at 4 °C. A sample from each purification step was resolved by SDS-PAGE  at 4 °C for 30 min. Then, they were diluted with 2 × Laemmli SDS sample buffer22.To reduce contamination and aggregates in the samples for SAXS measurements, SEC-SAXS was applied to the samples. In this measurement, a stopping mode in a laboratory-based SEC-SAXS system was adapted to increase the statistics of scattering intensity  and incubated at 37 °C for 2 h. The samples were resolved by SDS-PAGE  as a function of the sedimentation coefficient was obtained by fitting the time evolution of sedimentation data with Lamm formula28 using SEDFIT software29. The sedimentation coefficient was also normalized to the value at 20 °C in pure water . The molecular weight for each component was calculated using s20,w and the friction ratio f/f0.Next, AUC was conducted to determine the abundance of monomers, dimers, trimers, other aggregations, and contaminations in the sample solution. The AUC experiments were performed at 60,000 rpm at 25 °C with a sedimentation velocity method using Rayleigh interference optics . The weight concentration distribution c. There was one major peak around s = 3.5, and two minor peaks around s = 5.2 and s = 7.0 in the AUC profiles of both oxidized and reduced samples. This means that the sample solutions were highly purified, but there were very small aggregates (s = 5.2 and s = 7.0). The detailed values are listed in Supplementary Table Supplementary Fig. MALDI-TOF MS was used to confirm the aggregation numbers of the above-mentioned aggregates in the sample solutions. The samples and a saturated solution of sinapinic acid  in TA30 (30% acetonitrile in 70% of 0.1% TFA in water) were mixed at a ratio 1:9 and dropped on the ground steel MALDI plate . After drying and crystallisation of the droplets, the measurement was conducted on a microflexLT MALDI-TOF mass spectrometer  in positive ion mode. External calibrations were performed using protein standards II. Mass spectra data were recorded with flexControl and analysed using FlexAnalysis .s-value.Supplementary Fig. −1 through a capillary column  packed with resin . Two microliters of ER-60 solutions were loaded into the column by a sample injector  with a 2 µL sample loop. The samples were directly analysed using a micrOTOF II mass spectrometer with an electrospray ionisation (ESI) source . The pressure of the nebuliser gas, temperature of the dry heater, flow of the dry gas, and capillary voltage were set at 0.5 bar, 250 °C, 3 L min−1, and 4000 V, respectively. The mass spectra were obtained in the positive ion mode with a scanning range of m/z 500–10,000. ESI-L Low Concentration Tuning Mix  was used for internal calibration. Mass spectrum data were analysed using Bruker Compass Data Analysis 4.2.As the final sample check, the average mass values of the samples were measured rigorously using LC-ESI-TOF mass spectrometry to determine whether ER-60 samples were truncated or degraded. One hundred-fifty mM ammonium acetate (pH 6.8) was flowed at 2 µL minSupplementary Fig. q range (0.01–0.70 Å−1), where, SAXS measurements were performed using a NANOPIX . X-rays from a high-brilliance point-focused X-ray generator  were focused with a confocal mirror (OptiSAXS) and collimated with the lower parasitic scattering slit system, “ClearPinhole”. The scattered X-rays were detected with a two-dimensional semiconductor detector  with a spatial resolution of 100 μm. Two sample-to-detector distance (SDD) conditions, 1330 mm and 300 mm, were used to cover a wide q range measurement , the SEC-SAXS measurement with “stopping mode”22 was employed to remove the unfavourable aggregates from the sample solution. Then, the scattering data were collected for 1120 min (10 min × 112). In the higher-q range measurement , standard SAXS measurements were applied. The scattering data were collected for 500 min (10 min × 50).In the lower-Iwater = 1.632 × 10–2 cm−1). All reductions were processed with SAngler30. The low-q and high-q profiles were connected between q = 0.061 Å−1 and 0.064 Å−1.The obtained two-dimensional scattering patterns were converted to one-dimensional scattering profiles by radial averaging. Subsequently, one-dimensional scattering profiles were corrected for the intensity of the incident beam and the transmission. Then, the scattering profile of a protein in solution was obtained by subtracting that of the buffer. Finally, the unit of the scatter intensity was converted to an absolute scale by comparison with the scattering intensity of water  after the SAXS measurement. The amount of aggregates did not change before and after SAXS, indicating that there were no further aggregates due to radiation damage. This is also clear from the peak positions and areas on c listed in Supplementary Table Using the same procedure as Analytical ultracentrifugation 1, we identified the abundance of monomers, dimers, trimers, other aggregates, and contaminants in the sample solutions immediately after the SAXS measurements. Supplementary Fig. Using the same procedure as that used for MALDI-TOF MS 1, we confirmed the aggregation numbers of the above-mentioned aggregates in the sample solutions immediately after the SAXS measurements. Supplementary Fig. 20. Supplementary Fig. As indicated in Supplementary Table Finally, we obtained aggregation-free SAXS profiles for detailed structural analysis.31. In this model, each amino acid is represented by a single particle at the Cα atom position. The reference structure for the structure-based potential was prepared by replacing the mutated Ala60 of the crystal structure of ER-60 (3F8U) with Cys using the Phyre2 modelling server17. The C-terminal region of ER-60 was modelled using PyMOL32. In order to make the structural modulation, we also added both Debye-Hückel-type electrostatic interactions with an ionic strength of 150 mM and hydrophobic interactions to the standard AICG2+ potential33.We used the atomic-interaction-based coarse-grained model (AICG2+), which is the so-called “structure-based model” 133 and Leu365, which are located between the a–b and b′–a′ domains, respectively, and the region from Gln490 to Leu505, which is a flexible C-terminal region (Supplementary Figs. b–b′ domains were regarded as one domain. For these folded regions, the charge of each amino acid was determined based on the surface charges of the reference structure34.We regarded the three peptides as flexible loops: Gly8 MD steps at 300 K using the simulation software CafeMol35. The simulation snapshots were collected every 5000 MD steps and a total of 800,000 snapshots were recorded.We performed 40 individual Langevin dynamics simulations for 10Supplementary Information.


UBE3A and SNORD116 located within the 15q11–q13 region between these disorders and their subtypes and related these to the clinical phenotypes. The study cohort included 58 participants affected with a C15 imprinting disorder  and 20 typically developing controls. Semi-quantitative analysis of mRNA from peripheral blood mononuclear cells (PBMCs) was performed using reverse transcription droplet digital polymerase chain reaction (PCR) for UBE3A and SNORD116 normalised to a panel of internal control genes determined using the geNorm approach. Participants completed an intellectual/developmental functioning assessment and the Autism Diagnostic Observation Schedule-2nd Edition. The Dup15q group was the only condition with significantly increased UBE3A mRNA levels when compared to the control group (p < 0.001). Both the AS and Dup15q groups also had significantly elevated SNORD116 mRNA levels compared to controls . Both UBE3A and SNORD116 mRNA levels were positively correlated with all developmental functioning scores in the deletion AS group (p < 0.001), and autism features (p < 0.001) in the non-deletion PWS group. The findings suggest presence of novel interactions between expression of UBE3A and SNORD116 in PBMCs and brain specific processes underlying motor and language impairments and autism features in these disorders.Chromosome 15 (C15) imprinting disorders including Prader–Willi (PWS), Angelman (AS) and chromosome 15 duplication (Dup15q) syndromes are severe neurodevelopmental disorders caused by abnormal expression of genes from the 15q11–q13 region, associated with abnormal DNA methylation and/or copy number changes. This study compared changes in mRNA levels of  Cardinal features include a poor suck with failure to thrive, infantile hypotonia and hypogonadism. Food seeking and hyperphagia emerges at approximately 6 years of age, leading to morbidity if not externally controlled. Mild ID  is typical, frequently accompanied by compulsions, tantrums and skin picking5. AS is characterised by microcephaly, gait ataxia, seizures, ID, and absence of speech6. Dup15q is associated with variable cognitive impairment and motor delays. An overlapping feature between AS and Dup15q is the presence of seizures3.PWS was the first example of genomic imprinting identified in humans7. Loss of paternal gene expression from the chromosome 15q11–q13 region is the primary cause of PWS7, while the absence of the maternal gene expression in the same region is the primary cause of AS3. For PWS the lack of expression of key genes result from: (i) two deletion subtypes  in ~60% of cases; (ii) three maternal disomy subtypes in ~35% of cases; and (ii) two imprinting centre defects  in ~5% of cases8. Similarly, deletions from the maternally contributed chromosome 15 are the most common cause of AS (~70% of cases). Paternal uniparental disomy (patUPD) occurs in approximately 8% of AS cases and ICD in approximately 7% of cases3. Approximately, 10% of AS cases result from a mutation in the ubiquitin-protein ligase E3A gene (UBE3A). Both PWS and AS have a frequency of approximately 1 in 15,000 births9.DNA methylation and/or copy number changes on chromosome 15 are thought to cause PWS and AS specific phenotypes9. In contrast, paternal Dup15q has a less severe phenotype than maternal Dup15q10. Despite Dup15q being a cause of autism spectrum disorder (ASD), reported in 1–3% of ASD cases9, prevalence in the general population has not been well established, with one study reporting 1 in 14,000 in the general population11.Dup15q syndrome results from duplications or triplications of the PWS/AS imprinted 15q11–q13 region. Triplication typically arises through the presence of a supernumerary chromosome (isodicentric 15 [idic15]), while the duplication is caused by interstitial tandem duplication (int dup[15]). Hereafter, we use Dup15q to encompass these subtypes, unless otherwise stated. In maternal Dup15q, autism features are more common and severe, as compared to AS and PWS, with severity directly proportional to the number of maternal copies present12. For PWS, those with the typical 15q11–q13 deletions have been reported to have lower Verbal IQ (VIQ) scores than those with matUPD13. In addition, PWS individuals with the larger typical 15q11–q13 type I deletion involving chromosome 15 breakpoints BP1 and BP3 have been reported to have more behavioural problems, specifically self-injury and compulsions compared to those having the smaller typical 15q11–q13 type II deletion involving breakpoints BP2 and BP314. The larger type I deletion encompasses four extra genes , which may account for the additional clinical findings.While some genotype–phenotype correlations have emerged in each of the syndromes, primarily around the different molecular classes, there is a need for peripheral tissue biomarkers in humans as the phenotypes are highly variable in each disorder, and their specific subtypes do not fully explain this variabilityUBE3A expression in the brain as the primary cause of specific deficits in AS15. However, few studies have been performed in human peripheral tissues17. UBE3A is a key gene in neurodevelopment and is thought to be imprinted only in neurons, where it is expressed from the maternal allele in humans and mice15. Interestingly in other cell types UBE3A has bi-allelic expression18 and is thought to be consistently expressed from both alleles in different peripheral tissues15. The silencing of UBE3A on the paternal allele in neurons is thought to be regulated by a paternally expressed antisense transcript of UBE3A. This antisense transcript is part of the 3′end of SNRPN–SNURF transcript that comprises multiple small nuclear RNAs (snoRNAs)19. SnoRNA C/D box cluster 116 (SNORD116) is one of the snoRNAs that serves as a precursor to the antisense UBE3A, and through this process may regulate UBE3A silencing on the paternal allele16. Importantly, in PWS SNORD116 has been reported to be completely silenced, in neurons as well as other cell types in peripheral tissues, and this may contribute to phenotype severity17.For AS, mouse models have been used to implicate loss of UBE3A and SNORD116 mRNA levels  in peripheral blood mononuclear cells (PBMCs) using a highly sensitive and quantitative droplet digital PCR (ddPCR) method developed for analysis of gene expression. Expression changes for these genes in PBMCs are investigated between the chromosome 15 imprinting disorders, the different subtypes and typically developing controls. For AS and PWS subtypes genotype–phenotype studies are also described, with the focus on relationships between the UBE3A and SNORD116 mRNA levels in PBMCs and brain specific phenotypes including formal assessments targeting behavioural features and intellectual functioning. It was hypothesised that expression of these genes in PBMCs reflects immune processes in the brain related to gene expression in microglial cells (of the same cellular lineage) that support neuronal processes in the brain related to intellectual/developmental functioning and autism features in these disorders20.This study, for the first time, examines 22 were included as reference data for gene expression comparisons.This study comprised 58 individuals with a chromosome 15 imprinting disorder aged between 1 and 45 years. Twenty-seven individuals with PWS, 21 individuals with AS and 10 individuals with Dup15q were included in the study. The molecular classes for each syndrome are presented in the supplementary material Table . SamplesParticipants were recruited through Victorian Clinical Genetics Services; Hunter Genetics; the PWS Clinic at the Royal Children’s Hospital, Melbourne; Genetic Services of Western Australia; and various support organisations including the Foundation for Angelman Syndrome Therapeutics, the PWS Clinic at the Royal Children’s Hospital, Melbourne; the Prader–Willi Association of Australia. Inclusion criteria were that verbal participants be able to speak English and non-verbal participants to be exposed to English within the home. Exclusion criteria were having another genetic condition of known clinical significance, or having any significant medical condition .23.For all participants their diagnosis was confirmed using DNA methylation analysis, methylation-specific multiple ligation-dependent probe amplification and/or microarray testing for copy number changes within the 15q11–q13 region, as per standard diagnostic testing protocol24. RNA was extracted from 1 to 4 million viable PBMCs using RNeasy kit as per manufacturer’s instructions . RNA samples were diluted between approximately 10 and 20 µl/ng depending on the concentration, resulting in an average input of 100 ng per reverse-transcription reaction. Reverse transcription was performed using the High-Capacity cDNA Reverse Transcription Kit as per manufacturer’s instructions .At the time of assessment, 5 ml of venous blood were collected in ethylenediaminetetraacetic acid tubes, followed by PBMC isolation using Ficoll gradient separation, as previously describedEIF4A2, RPL13A, SDHA, TOP1, YWHAZ, APT5B and GAPDH) were used from the geNorm reference 12 gene kit, as per manufacturer’s instructions . For each assay, 1.1 µl of the primer/probe mix was added to 1× ddPCR SuperMix for probes (no dUTPs)  and RNAse/DNAse free water, and then aliquoted into 96-well PCR plates. One to two microlitre of cDNA was then added to the reaction mix.To establish the optimal reference genes for our samples, eight assays (SNORD116 used the forward (5′-CAGGAAAGATCAAAACGATGCA-3′) and reverse (5′-TCCAAAGGAGGCAGTTGGAT-3′) primers with a HEX labelled probe (5′- TGCAAGTGTGATTGGTCCAGATAGCTGC-3′). This target region was referred to as SNURF–SNRPN exons 57–58, in a previous study16. The UBE3A assay was a premade TaqMan Assay (ID: Hs00166580_m1) with specific primer sequences not made available by the manufacturer . The assay targets an amplicon with Chr.15: 25337234–25439086 coordinates [on Build GRCh38] present in most UBE3A transcripts, utilising a FAM-labelled probe. Both UBE3A and SNORD116 assays utilised a Deep Hole Quencher and were duplexed. Briefly, each 20 μL reaction mixture contained 2.0 μl cDNA, 1× ddPCR SuperMix for probes, no dUTPs , 125 nM SNRPN forward primer, 125 nM SNRPN reverse primer, 500 nM SNRPN HEX tagged probe, 1× UBE3A FAM-tagged TaqMan assay, and RNAse/DNAse free water. Prepared reactions were run on a Bio-Rad QX200 system . Data were analysed using the QuantaSoft Analysis Pro software, as per manufacturer’s instructions , with data collected from two different channels specific for HEX and FAM signals .For target genes, 25, a developmental assessment that assessed the domains of visual reception, fine motor, receptive language, and expressive language. Developmental assessments are commonly used for individuals with AS and Dup15q, given the significant developmental delay that is commonly observed in these individuals and the subsequent propensity to have scores that fall at the “floor” on standardised age-appropriate assessments. The MSEL provides age equivalent scores in months for the domains of visual reception, fine motor, receptive language and expressive language. All PWS participants completed an age appropriate standardised assessment. Standardised intellectual functioning assessments included the MSEL (children aged under 3 years), the Wechsler Preschool and Primary Scale of Intelligence-3rd Edition 26, the Wechsler Intelligence Scale for Children-4th Edition Australian 27 or the Wechsler Adult Intelligence Scale-4th Edition Australian edition 28. The following intellectual functioning variables were included in the analyses: VIQ, performance IQ (PIQ) and FSIQ.All individuals with AS were assessed with the Mullen Scales of Early Learning (MSEL)29 was used to assess social communication skills and the presence of repetitive and restricted behaviours. The ADOS-2 was used in all children who were cruising/walking and had a mental age of 12 months or greater. The ADOS-2 provides an overall calibrated severity score (CSS), in addition to CSS for the social affect (SA CSS) and repetitive and restricted behaviours (RRB CSS) domain. All assessments were completed by the first author, a certified ADOS-2 trainer in Australia, who regularly engages in reliability coding meetings to maintain research reliability of >80%.The Autism Diagnostic Observation Schedule-2nd Edition (ADOS-2)Participants attended an appointment at a clinic or within their own home for the developmental/intellectual functioning assessment and the ADOS-2. Venous blood was collected at the cessation of the clinical appointment. All procedures were approved by The Royal Children’s Hospital Human Research Ethics Committee (HREC #33066). All parents/caregivers provided written informed consent, and those aged 18 years and above, who were deemed cognitively able, also provided written consent.UBE3A and SNORD116 mRNA copy numbers were normalised to copy numbers from a stably expressed set of internal control genes determined using the geNorm approach30 on the same set of cDNA samples from PWS (n = 21) and control (n = 23) cohorts . The Bonferroni correction method was used to correct for multiple testing. All analyses were conducted using the software STATA (http://www.stata.com).UBE3A and SNORD116 . Similarly, the proportion of males in each group did not significantly differ . Both the Dup15q and PWS groups had a significantly higher ADOS-2 CSS and SA CSS compared to the AS group  = 7.15, interquartile range (IQR) = 12.31; AS: Md = 6.93, IQR = 16.03; Dup15q: Md = 6.38, IQR = 9.83; UBE3A mRNA levels compared to the PWS, AS and control groups . For the Dup15q group UBE3A mRNA levels (range: 0.268–0.574) were significantly more variable than those for SNORD116 mRNA (range: 0.026–0.052). In contrast SNORD116 mRNA was completely absent in the PWS group, but the levels of UBE3A mRNA were not significantly different between the PWS and control groups  than those found in the deletion subtype  scores derived from the MSEL  to determine the optimal combination of most stably expressed genes, to be chosen as appropriate reference controls. This is a significant limitation of the earlier studies32 that needs to be taken into consideration, when interpreting and comparing results from this study with those conducted earlier.This study was also the first to utilise ddPCR, as a more precise and reproducible technique to quantify 32 that are not as precise or quantitative as ddPCR for locus specific analysis33; (ii) target gene(s) normalised to a single internal control gene such as β-actin16, β-2 microglobulin15 or GAPDH32 that were not validated as part of gene expression stability studies, and may be themselves affected in the tested conditions. In this study, SDHA, ATP5B and EIF4A2 were identified as the combination of reference genes most stably expressed in these settings, and were used to normalise expression of target genes, UBE3A and SNORD116. Future studies aiming to examine expression of other genes in these settings, now have evidence supporting use of the same normalisation strategy as described here for UBE3A and SNORD116.Together, our approach addresses two potential technical limitations associated with previous gene expression studies in chromosome 15 imprinting disorders including: (i) use of techniquesSNORD116 mRNA levels showed the greatest variability in the AS group as compared to UBE3A. However, for UBE3A there was almost a clear split, with the deletion AS individuals showing lower mRNA levels than the non-deletion group, with the exception of one individual with AS and the deletion subtype, who had one of the highest levels of expression reported for all individuals examined. For SNORD116 the UBE3A mutation group appeared to cluster with the deletion subtype. This suggests that there is a dissociation between UBE3A and SNORD116 expression in AS, at least in peripheral blood; a similar phenomenon to what was observed in blood of PWS and AS patients previously16.The UBE3A was the highest from all groups examined with significant variability observed. As expected, those with the int dup15 subtype had the lowest UBE3A mRNA levels within the Dup15q group. Moreover, the results for the combined Dup15q group were approximately four times those of controls. This is consistent with UBE3A not being imprinted in PBMCs, and further demonstrates the reliability of the ddPCR method developed and used to quantify mRNA levels in PBMCs in this study. However, this does not necessarily rule out that factors other than copy number changes impact mRNA levels in Dup15q individuals with the idic15 subtype, consistent with results previously observed in idic1534. In contrast, SNORD116 mRNA levels were less variable within the Dup15q group, with expression being not significantly different from the AS group, but still significantly above the control levels. Since SNORD116 in PWS is completely silenced in PBMCs on maternal alleles, one would expect to see mat idic15 to have SNORD116 expressed at normal levels . The observed elevated levels, however, suggest that SNORD116 on the paternal allele is being upregulated in PBMCs, though the mechanism is yet to be characterised. These findings should be confirmed in a larger independent cohort, as elevated levels of SNORD116 mRNA from the paternal allele may contribute to mat idic15 pathology.For the Dup15q group, the expression of 34, PWS-IC methylation, and UBE3A transcript and protein levels were higher in Dup15q than in control or autism samples. Methylation of the PWS-IC region showed a positive correlation with UBE3A and GABRB3 levels, but a negative correlation with SNRPN levels. This led the authors to suggest that gene expression within 15q11–q13 is not completely based on copy number changes, but it can also be influenced by epigenetic mechanisms in the brain.Moreover, in a study describing methylation-sensitive high-resolution melting-curve analysis on genomic DNA of post-mortem human brain tissues obtained from 8 Dup15q syndrome, 10 idiopathic autism and 21 typical control individualsUBE3A mRNA levels were not significantly different compared to controls, while SNORD16 expression was completely silenced. This was consistent with the real-time PCR results from an earlier study of SNORD116 (referred to as SNRPN exons 57–58)16 that used exactly the same primers and probe as in this study. The previous study tested blood of two PWS participants, and also did not detect any SNORD116 mRNA levels by real-time PCR. For quantitative analysis in AS and controls, however, it is not possible to accurately compare the results between the studies16 because: (i) the previous study reported quantitative real-time PCR results as a ratio of SNORD116 over UBE3A mRNA, rather than normalising expression of each of the target genes to a set of stably expressed internal control genes; (ii) the number of participants was too low to perform statistical analysis as part of intergroup comparisons.In PWS (regardless of the molecular class) UBE3A and SNORD116 mRNA levels in PBMCs, genotype–phenotype relationships were examined. Both UBE3A and SNORD116 mRNA levels in PBMCs were significantly correlated with all age equivalent scores on the MSEL and the RRB CSS in the deletion AS group; higher gene expression levels were associated with better developmental functioning and less repetitive and restricted behaviours in this subtype. In the PWS deletion group, higher UBE3A mRNA levels were associated with lower SA CSS scores. In contrast, for the PWS non-deletion subtype, increased UBE3A mRNA levels were significantly associated with lower PIQ scores and increased SA CSS, reflecting poorer visuo-spatial and social communication skills, respectively. No significant associations were observed between UBE3A and ADOS scores in the Dup15q group. This may be attributed to the highly heterogenous molecular and clinical profiles of individuals with Dup15q and a significantly smaller sample size than the AS and PWS groups with available ADOS-2 data. Nonetheless, all Dup15q participants who completed the ADOS-2 (n = 8) met the cut-off for ‘Autism’ indicating a high level of autism symptomsTo examine potential clinical and biological significance of the detected variation in UBE3A has been linked to severity in Dup15q, where the increased number of maternal alleles is thought to be the primary driver of Dup15q pathology34. Similarly, PWS individuals with the UPD subtype are also thought to be ‘at a greater risk’ for ASD due to overexpression of UBE3A35. The findings provide support to this theory, however, it is possible that overexpression of UBE3A is more specifically related to the social communication deficits associated with ASD rather than RRBs. Several of the findings in this study support this notion: (i) higher UBE3A levels were associated with lower SA deficits in the PWS deletion subtype, which clustered at the bottom of the PWS group on UBE3A levels; (ii) in those with the non-deletion subtype higher UBE3A levels were associated with more SA difficulties; (iii) RRBs did not significantly differ across the three clinical groups, despite differences in UBE3A expression. Moreover, in ASD and other psycho-pathologies, such as schizotypy, gain of function UBE3A mutations have been suggested to be linked to phenotype severity38. It should be noted that ADOS scores in individuals with PWS may also be capturing other psychiatric difficulties including mood disorders and schizotypy traits, which are also linked to social communication deficits39. There is now a large amount of evidence supporting UBE3A dosage effects on schizotypy in the general population38 as well as those with PWS and Dup15q.Overexpression of UBE3A and SNORD116 mRNA levels and brain related processes, including developmental functioning and autism features, are consistent with the patho-mechanism proposed in a recent study examining transcriptomic signatures in the hypothalamus from post-mortem brain tissues in PWS40. This previous study demonstrated that genes associated with inflammatory response  were upregulated in PWS hypothalamus40. Together with the genotype-phenotype relationships reported in this study using PBMCs, this suggests that the immune response regulated by microglial cells  may play a broader role in chromosome 15 imprinting disorder pathology. Abnormal expression of UBE3A in AS leucocytes, where UBE3A is not imprinted, may also have implications for clinical trials targeting UBE3A re-activation in the brain41. Specifically, UBE3A expression and regulation in other tissues may need to be monitored for non-specific impacts of the intervention on pathways regulated by UBE3A in non-neuronal cell types. Moreover, future studies should explore UBE3A expression in microglial cells in post-mortem human tissues of individuals affected with different chromosome 15 imprinting disorders, to confirm or refute these proposed patho-mechanisms.The associations between 33. Another strength is the characterisation of the most stably expressed internal control genes  using the geNorm approach in PBMCs of PWS and control groups30. This approach to select stably expressed reference genes for quantitative real-time PCR30 and ddPCR42 based studies has been widely recommended in other settings to ensure that the variation identified is not due to changes in genes used for target gene normalisation. Yet, previous studies for locus specific gene expression analysis in the chromosome 15 imprinting disorders in peripheral tissues and/or neurons, mostly utilised expression of β-actin16, β-2 microglobulin15 or GAPDH32 as a single internal control gene. Of note is that β-actin and β-2 microglobulin are involved in cell migration, which may be affected in chromosome 15 imprinting disorders43.One of the strengths of this study is the utility of ddPCR to accurately measure differences in gene expression of 30% or less—a feat not possible with most other methods including real-time quantitative PCR and RNA-seq44. Given these are rare genetic syndromes, obtaining large sample sizes can be difficult for each subtype. Nonetheless, this is one of the largest studies to date that has examined the utility of gene expression in peripheral tissues as biomarkers for brain related phenotypes in chromosome 15 imprinting disorders. Another limitation is that cellular and secreted protein product of UBE3A was not examined in this study. This is an important direction for future studies to confirm functional significance of changes in UBE3A mRNA levels and relationships with clinical phenotypes reported here.A limitation of the current study is the relatively small sample sizes for each of the subtypes in each condition as compared to similar studies performed in other rare disorders such as fragile X syndrome (FXS)44. Nonetheless, future research exploring relationships between peripheral tissue biomarkers and other tissues such as CSF and brain tissue, where available, will further confirm the utility of peripheral biomarkers in understanding the molecular underpinnings of these rare genetic syndromes.While the use of peripheral tissues may not be entirely reflective of what is happening in the brain, the primary advantage of examining peripheral tissues is that the collection is less invasive than other tissues, such as cerebral spinal fluid (CSF), and will therefore likely increase sample sizes which is important for recruitment of individuals with rare genetic disorders. Evidence from other neurodevelopmental disorders, including FXS, has demonstrated significant relationships between promoter methylation, mRNA and protein and gene expression in peripheral tissues  and the severity of the clinical phenotype, including intellectual functioning and autism featuresUBE3A mRNA levels in PBMCs were shown to be significantly correlated with brain related phenotypes in both the PWS and AS cohorts. Specifically, increased UBE3A mRNA levels correlated with increased autism features in PWS and receptive language skills in AS, suggesting the utility of gene expression analysis in peripheral tissues as non-invasive biomarkers for future studies. Future research exploring clinical and biological significance of the relationships between UBE3A and SNORD116 transcription in human peripheral and post-mortem brain tissues in PWS, AS and Dup15q patients are warranted to further explain the findings and the underlying mechanisms proposed in this study.In conclusion this study has demonstrated the utility of ddPCR for accurate quantification of gene expression in PBMCs in chromosome 15 imprinting disorders and characterisation of a stably expressed set of internal control genes. This ensured that variation reflected in the target genes analysed is not attributed to the variability in expression of the internal control gene/s used. Supplemental TablesSupplemental Figure legendsSupplemental Figure 1Supplemental Figure 2


Our approach first enables Domain Confusion (DC) by using a domain discriminator for adversarial training. For Classifier Adaptation (CA), a residual block is added to the source domain classifier in order to learn the difference between source classifier and target classifier. Beyond validating our framework on the standard domain adaptation dataset office-31, we also introduce and evaluate on the Comprehensive Cars (CompCars) dataset, and the experiment results demonstrate the effectiveness of the proposed framework C2DAN.Deep neural networks have been successfully applied in domain adaptation which uses the labeled data of source domain to supplement useful information for target domain. Deep Adaptation Network (DAN) is one of these efficient frameworks, it utilizes Multi-Kernel Maximum Mean Discrepancy (MK-MMD) to align the feature distribution in a reproducing kernel Hilbert space. However, DAN does not perform very well in feature level transfer, and the assumption that source and target domain share classifiers is too strict in different adaptation scenarios. In this paper, we further improve the adaptability of DAN by incorporating Domain Confusion (DC) and Classifier Adaptation (CA). To achieve this, we propose a novel domain adaptation method named C In recent years, deep learning has made great achievements in a large number of computer vision tasks, such as image recognition ,2,3, objThe main idea of domain adaptation is to reduce the difference between domains and to learn a prediction model. Two types of mechanism are used most to address domain shift: discrepancy-based and adversarial-based. The representative discrepancy-based methods ,18,19,20However, these methods are not without problems. The typical discrepancy-based Deep Adaptation Network (DAN) framework  which is2DAN which improves the DAN framework by combing it with Domain Confusion (DC) and Classifier Adaptation (CA). These changes make the extracted feature representation more adaptive to the target domain. At the same time, aiming at solving the problem of different adaptation effects of MK-MMD in different scenarios, this paper explores the suitable weight selection of MK-MMD. In addition, we pursue the best combination of MK-MMD, DC and CA to further enhance the efficiency of domain adaptation. Experiments on the standard domain adaptation dataset Office-31 [2DAN. In this paper, inspired by the adversarial-based method  and Domaffice-31  dataset (1)We combine Domain Confusion (DC) with MK-MMD in DAN for both feature alignment and domain alignment, which makes the model more generalized in the target domain.(2)The model is extended by adding Classifier Adaptation (CA) to minimize the difference of source classifier and target classifier, the accuracy of the proposed method is further improved.(3)2DAN surpass the performance of DAN.The best combination of MK-MMD, DC and CA in different scenarios is obtained through experiments on office-31 and CompCars dataset, the experimental results show that our improved method CThe contributions of this paper are as follows:The structure of this paper is as follows: For the deep domain adaptation methods, the basic criterion is to add the adaptation metric after selecting the adaptive layer, and then to fine tune the network. At the PRICAL meeting in 2014, the Domain Adaptive Neural Network (DaNN) concept  was put Adversarial-based models use a generator to align the source and target domain data in feature space by working against the discriminator. Generative Adversarial Network (GAN)  has beenIn this work, the DAN  method i2DAN is shown in The network structure proposed in this paper is based on DAN  which ads and t is calculated as follows:s, then an empirical estimate of MMD can be expressed as follows:L basis kernels Multi-kernel Maximum Mean Discrepancy (MK-MMD) is an extension of MMD. MMD is one of the most commonly used non-parametric methods to measure the distribution difference between two domain datasets. The detailed operation is to map the feature representation of source domain and target domain to the reproducing kernel Hilbert space (RKHS), and then calculate the mean distance between the two datasets in RKHS. Given the data distribution s and t of the two domains, and by using the function kH denote the RKHS with the characteristic kernel k, and let the mean value of distribution p in kH be an independent element p and q is expressed as The MMD method is based on single kernel transformation. The multi-kernel MMD (MK-MMD) assumes that the optimal kernel can be obtained by linear combination of multiple kernels. One of the most successful ways to use MK-MMD is DAN. As shown in , let Hk ϕ is related to characteristic kernel, so m PSD kernels k can use different kernels to enhance the effect of MK-MMD and achieve an optimal and reasonable kernel selection:Similar to MMD, feature mapping k is weighted by different kernel and the coefficients k is characteristic.In order to reduce the difference of marginal distribution between two domains, maximizing domain confusion (DC) is an effective method. The schematic diagram of domain confusion is shown in q represents the softmax value of the domain classifier, According to , for a fThe optimization of the domain confusion loss above is to find a domain invariant feature representation, in which the best domain classifier cannot achieve good results. Ideally, we want to optimize these two losses simultaneously in the training process. But there are two contradictions between domain classification and domain confusion. Learning a good domain classifier means that the effect of domain confusion is very poor, and also reaching a good domain confusion effect means that the effect of domain classification is very poor. Therefore, the two losses of layer In order to reduce the difference of classifiers between two domains, adding classifier adaptation (CA) is also an effective method. CA is motivated by the deep residual learning  which is(x)+x in , we can J is the cross-entropy loss function, and H is the entropy loss function, c is the number of classes and j for Although classifier adaptation (CA) has reduced the difference of source classifier and target classifier, the output of target classifier L learns a nonlinear mapping Lth layer, Lth layer, and According to DAN’s network, this paper uses the classic five convolutional layers and three full connection layers. Each full connection layer In this paper, we add classifier adaptation (CA) by adding two fully connected layers as residual layers to ensure that λ, γ and β are the balanced weights of MK-MMD, DC and low-density separation between classes of target domain respectively.In the standard convolutional neural network, the deep feature representation becomes from generalization to specialization with the layer from low to high. The higher the network layer is, the greater domain’s difference is, which cannot be reduced by fine-tune alone. Therefore, it is necessary to make adaptation on the full connection layers instead of convolutional layers. In this paper, we propose two ways to do domain adaptation on the full connection layers and reduce the difference of domain distribution. One way is to add MK-MMD based multi-layer adaptation, and the other way is to add a domain classification layer To train a convolutional neural network from scratch requires a large amount of labeled data, which is also impossible for domain adaptation problems, so in this paper the initial network adopts AlexNet model trained on ImageNet 2012 as the pre-trained model. Then we freeze the convolution layer from layer 1 to layer 3 and fine-tunes the 4th and 5th convolution layer, and other parameters are calculated by optimizing the loss function through training.For different domain adaptation scenarios, the weight of loss function could be different. When we combine MK-MMD, domain confusion and classifier adaptation, we also explore the contribution’s difference with different domain adaptation metric to different scenarios. We also consider the reasonable combination of the two methods. This part is explained in Office-31 [Office-10+Caltech10 dataset contains 10 categories which are selected from Office-31 and Caltech-256. There are four domains in total, three domains  separated from Offce-31 and one domain (C) from Caltech-256. Then six transfer tasks can be built: A –> C, W–> C, D –> C, C –> A, C –> W and C –> D. While Office-10+Caltech has less categories, it will be easier to make adaptation than Office-31 dataset. In this paper, experiments and comparative analysis will be conducted on the two datasets.In addition to the improved deep adaptation network which combines DC and MK-MMD proposed in this paper, the experiment will be compared with the traditional CNN method, the DAN method only using MK-MMD and the Residual Transfer Networks (RTN) method using classifier adaptation (CA). At the same time, in order to explore the application value and conditions of domain confusion in this network, the experiments of FC6 and FC7 based on DAN and Combination of RTN and DC also will be conducted and compared. Finally, we do the experiments which combining MK-MMD, DC with CA.In this paper, we use Caffe to develop the proposed deep neural network, which contains the classic five convolutional layers and three full connection layers. Similar to the method of DAN, MK-MMD layers are attached to the three full connection layers to calculate the multi kernel maximum mean discrepancy of feature representation between source domain and target domain in Hilbert space. At the same time, the domain classification layer 2DAN based on the mmd weight and entropy weight. The detail to choose weights is introduced in Here we use fine-tuning method to train this model. Considering the problem of the amount of data in the training set, we get the parameters of the first three convolutional layers from the pre-training model and freeze them. Then we fine-tune the subsequent convolutional layers and the full connection layers by the way of back-propagation. The random gradient descent method is used in the experiment, which parameter is 0.9. And the learning rate adopts ‘inv’ method, which parameter is 0.75, and the initial learning rate is 0.001. We set the batch size to 64 for all methods, and optimize the learning rate for each model independently. Since all the comparative methods use mmd as test statistic, Gaussian kernel is used to median pairwise squared distances on training data. We perform cross-valuation on labeled source data to select candidate parameters, the weights are selected following the strategy: (1) Try to get the best results by adjusting the mmd weight when we apply the DAN method. Then the mmd weight is fixed, which means we use the same mmd weight in all other methods. (2) Try to get the best results by adjusting the entropy weight when we realize the RTN method and fix the entropy weight. (3) Realize our method C2DAN combining DC, CA and MK-MMD, which means DAN+DC+CA, on the Office-31 dataset and Office–10+Caltech-10 dataset. And comparing with the ordinary convolution neural network CNN (Baseline), the classical deep adaptation network DAN [This experiment implements the unsupervised domain adaptation of the improved deep adaptation network Cwork DAN  and RTN work DAN , the expFrom the experimental results in 2DAN, we show the results of each module of C2DAN in 2DAN, the average accuracy of C2DAN is 73.8% on the Office-31 dataset, which is 0.6% higher than RTN+DC, but the accuracy of C2DAN is similar to RTN+DC in Office-10+Caltech-10, which seems little improvement. The difference between the two datasets is Office-10+Caltech-10 has less categories which make it easier to be transferred. Both of these two methods have combined domain confusion with classifier adaptation, the difference is C2DAN conducts feature alignment by MK-MMD and RTN by tensor MMD. From the data point of view, the combination of MK-MMD, DC and CA is more effective in more difficult tasks.To go deeper into different modules of CIn this paper, we explore and analyze the suitable weights of three different domain adaptation loss. The weight parameter of MK-MMD ranges from {0.1, 0.4, 0.7, 1, 1.4, 1.7, 2}, the weight parameter of domain confusion ranges from {0.01, 0.05, 0.1, 0.15, 0.2} and the weight parameter of classifier adaptation ranges from {0.05, 0.1, 0.15, 0.2, 0.25} It’s necessary to select appropriate weights and best combination in different scenarios.λ is 1, as shown in λ is 0.1. Therefore, in the actual application, the weight of MK-MMD λ should be 1 in the case of abundant information in the source domain, such as website images. Then, the weight λ should be 0.1 in the case of less information and complex environment in the source domain, such as images captured by web camera or digital SLR camera.For the weight parameter of MK-MMD, as shown in , the adaλ for every adaptation scenario. When the source domain is A and λ is 1, the best result can be obtained if γ equals 0.1. The comparison of different weights γ from domain A to domain W is shown in λ is 0.1, the best result can be obtained if γ equals 0.01. The comparison of different weights γ from domain W to domain A is shown in For the weight parameter γ of λ and γ for every adaptation scenario. When the source domain is A and λ is 1 and γ equals 0.1, the best result can be obtained if β equals 0.2. The comparison of different weights β from domain A to domain W is shown in λ is 0.1 and γ is 0.01, the best result can be obtained if β equals 0.05. The comparison of different weights β from domain W to domain A is shown in For the weight parameter β of entropy loss, we should consider the optimal combination of the three weight parameters. In this paper, we conduct experiments and analyze to get the best fit β under the optimal From the experimental results, as a domain adaptation method, the weight of domain confusion loss should be one tenth of MK-MMD loss. In theory, domain confusion aims to make the most distinguishable feature representation get the worst classification effect. If the weight of In summary, when domain adaptation is carried out from the domain with rich obvious information (e.g. network images) to the domain with less complex information , the weight of MK-MMD is 1, the weight of domain confusion is 0.1 and the weight of classifier adaptation is 0.2. For reverse domain adaptation scenario, the weight of MK-MMD is 0.1, the weight of domain confusion is 0.01 and the weight of classifier adaptation is 0.05. The weights above make the combination of MK-MMD, domain confusion and classifier adaptation more reasonable and lead to better result.This paper uses Comprehensive Cars (CompCars)  dataset Each section contains 163 types of vehicles, each of which is subdivided into smaller categories according to specific models and years. The model of the experiment is based on AlexNet. The model level is not deep enough and the capability is limited. Therefore, 20 kinds of vehicles are selected for unsupervised domain adaptation experiments, instead of using the whole dataset. The examples of website vehicle image dataset in CompCars are shown in From the above images, we can see that the shooting angle of the vehicle in the website dataset is not the same. The front, side and back angles are all included. But in the surveillance images, the vehicles are taken from the front with only slight angle change. The angle of view of side and back has changed too much and the distribution is totally different, so this kind of images is not suitable for domain adaptation training. Therefore, we need to filter the website images and only select the pictures taken from the front and the side view which include the vehicle head information.The examples are shown in From Details of the dataset used in this experiment are as follows. The website dataset(data) contains 20 categories, totaling 6425 pictures, and the surveillance dataset(sv_data) contains 20 categories, totaling 6960 pictures. The categories and their quantities are shown in the We use four methods to conduct comparative experiments: convolutional neural network CNN (baseline), DAN, the proposed deep adaptation network combining domain confusion with MK-MMD and the combination of DC, CA and MK-MMD. According to the analysis result, the experiment adapts from the domain with rich information network images to the domain with surveillance images including less rich information. Therefore, in our experiment, the loss weight of MK-MMD is 1, the loss weight of domain confusion is 0.1, and the loss weight of classifier adaptation is 0.1.The experiment is carried out under the framework of caffe. We need to add MK-MMD layer, domain confusion layer and residual layers, and then recompile caffe. Twenty types of vehicles are labeled with "0" ~ "19". During the training process, there is no label information in the vehicle image of the target domain. The accuracy results of vehicle classification experiments are shown in First of all, we need to point out that although the CompCars dataset is used for fine-grained classification, the purpose of our experiment is to verify the ability of unsupervised domain adaptation method to solve the model degradation problem when the source domain training model is applied in the target domain (without any labels). The emphasis is on the improvement of the domain adaptation effect, so it is inappropriate to compare with advanced fine-grained classification model.Meanwhile, it can be seen from the examples of website images and surveillance images that the source and target images used in this experiment have various changes in background, illumination, angle of view, etc. The domain shift is large, so the overall accuracy of this experiment is not high. However, even in this complex case, the method in this paper has achieved better results. In the follow-up study, assuming that the available source and target domain data are purer or more task-specific, the domain adaptation effect of this method will be better. The experimental results are analyzed as below.2DAN is 50.7%, 5.8% higher than DAN and 6.4% higher than RTN. It can be in inferred that the classification accuracy is greatly enhanced because the huge difference between source domain and target domain has been reduced by CA and DC. The results demonstrate that the combination of MK-MMD, DC and CA is reasonable and efficient. Moreover, comparing with the standard domain adaptation datasets Office-31 and Office-10+Caltech-10, the improvement of C2DAN on CompCars is more significant, which means C2DAN has powerful adaptation ability on challenging tasks. From the experimental results, it can be seen that the accuracy of unsupervised domain adaptation in vehicle classification task from the labeled source domain to the unlabeled target domain is 35.1% without any domain adaptation components. The DAN method based on the MK-MMD component has a classification accuracy of 44.9%, which is 9.8% higher than that of baseline. This proves that the domain adaptation can overcome inter-domain distribution differences caused by the variation of illumination and angle of view. The classification accuracy of DAN+DC and RTN+DC are both higher than the original methods DAN and RTN, which can prove the effectiveness of DC. The accuracy of our method CHere, we compare and analyze the classification accuracy of each vehicle type. The accuracy results of each vehicle type are shown in the following (1)2DAN methods. Compared with that of the DAN method, only one type has decreased, the data in the table show that the difference is little and the decline is not serious. The accuracy of 85% vehicle types have been improved, which proves that the proposed method is reasonable and effective.It can be seen that the classification accuracy of 17 types of vehicles has been improved after using the proposed DAN+DC or C(2)For vehicle types with less data, such as the Besturn and Dongfengfengdu cars, in which type the number of samples in source domain and target domain are both less than 2% of the total number of datasets, the proposed method improves the accuracy of vehicle classification by 28.5% and 5.5% respectively compared with the DAN method, and improves by 38.7% and 13.1% compared with the method using only CNN. It proves that the proposed method’s superiority is obvious. The feature extracted by the model in a limited number of samples greatly improves the representation ability in the target domain. By enhancing the feature invariance, the distribution difference between the source domain and the target domain is further reduced.(3)For the performance degradation of some classes, the main reason is that MK-MMD is an active acquisition while domain confusion is a passive verification for domain invariant feature. Besides, when the domain invariant property of the feature extracted by MK-MMD has reached the optimum level, the improvable space is very limited and the balance training of two kinds of loss may sacrifice the ability of domain adaptation in some categories. However, the sacrifice degree of this part is not too large. It is within the acceptable range and the accuracy of most categories has been improved.From the results above, the following conclusions can be drawn:Generally speaking, the proposed method indeed improves the effect of domain adaptation in the whole vehicle classification dataset.2DAN, which combines MK-MMD with Domain Confusion (DC) and Classifier Adaptation (CA). DC is added to learn domain invariant features by using a domain classification layer to perform adversarial training. We add residual blocks in source classification layer to preform CA which can reduce the discrepancy between source classifier and target classifier.This paper proposes an improved deep adaptive network C2DAN shows great advantages to reduce the huge differences between two domains. It can be inferred that DC is helpful to align the feature distribution with the use of MK-MMD. CA is also necessary especially in difficult transfer scenarios, we can’t use shared classifier to get good performance because the classification standards of source domain and target domain could be quite different.Experiments on standard domain adaptation datasets Office-31, Office-10+Caltech-10 and vehicle classification dataset CompCars show that: (1) DC and CA are both efficient to reduce the domain shift, which means the proposed method can improve the adaptation ability when knowledges are transferred form labeled source domain to unlabeled target domain. (2) The most suitable weights of MK-MMD and the best weight combination of MK-MMD, DC and CA in different transfer scenarios are explored from theory and experiments. (3) When used in challenging transfer tasks, CIn the future work, domain adaptation is going to face challenges that are harder than ever, DC and CA are both illuminating ideas which can be added to new domain adaptation methods. Furthermore, our work is expected to be applied in other computer vision tasks such as fine-grained classification and object detection.