Detailed analysis of the sleep regulatory mechanism shows that 14-3-3ε directly targets the Ultrabithorax (Ubx) gene to activate transcription of the pigment dispersing factor (PDF). The dopamine receptor (Dop1R1) and the octopamine receptor (Oamb), are also involved in the 14-3-3ε pathway, which in 14-3-3ε mutant flies causes increases in the dopR1 and OAMB, while downregulation of the DopR1 and Oamb can restore the sleep phenotype caused by the 14-3-3ε mutation. In conclusion, 14-3-3ε is necessary for sleep regulation in Drosophila.Sleep is a crucial factor for health and survival in all animals. In this study, we found by proteomic analysis that some cancer related proteins were impacted by the circadian clock. The  Drosophila melanogaster, which is detected by measuring the activity through a Drosophila activity monitoring (DAM) system  × 65[L] mm) with fly food. Experiments were performed in a Climate box at 25 ± 1 °C with 50% relative humidity. Light was turned on at ZT0 (06:30) and off at ZT12 (18:30). The activity data were recorded using the . (2013) . The flies were immobilized in 4% paraformaldehyde for 12 h at 4˚C and were then dissected in phosphate-buffered saline (PBS). The brains were blocked in blocking buffer  at RT for 2 h. The tissue was incubated in primary antibody for 24 h at 4 °C before being incubated with secondary antibodies overnight at 4 °C. The primary antibodies were as follows: mouse anti-PDF , rabbit anti-GFP , and mouse anti-RFP . Fluorescent secondary antibodies conjugated to Goat anti-Rabbit FITC  and Goat anti-Mouse TRITC . The immunofluorescence assay was carried out on a Leica system .TM RT reagent Kit with gDNA Eraser (TaKaRa Japan Cat# RR047A). A total of 1μL cDNA was added in each real-time PCR reaction system. SYBR Green method was used for Real-time PCR with SuperReal PreMix Plus kit (TIANGEN China Cat# FP205-02). The PDF gene real-time PCR program: holding stage 95 °C 10mins; cycling stage 95 °C 15 s, 57 °C 25 s, 68 °C 35 s, 40 cycles; melt curve stage 95 °C 15 s, 60 °C 1 min, temperature increment +0.3 °C, 95 °C 15 s. The Ubx gene real-time PCR program: holding stage 95 °C 10 min; cycling stage 95 °C 15 s, 60 °C 20 s, 72 °C 30 s, 40cycles; melt curve stage 95 °C 15 s, 60 °C 1 min, temperature increment +0.3 °C, 95 °C 15 s. RP49  was regarded as reference gene. 1118w control was used for normalization. The △△CT method was used for quantification. The real-time PCR data analysis is described in Livak et al. (2001) [Total RNA was isolated from heads of five to seven-day-old flies using RNAiso plus (TaKaRa Japan Cat# 9109). Each sample contained 30 individual flies with three biological repeats, which were reversely transcribed and measured by real-time PCR, respectively. There were three technical repeats in each biological repeat in the real-time PCR experiment. The total RNA quality was checked by Agilent Bioanalyser. A total of 1 μg RNA was added in each reverse transcription system. The RNA was reversely transcribed with a PrimeScrip. (2001) . The reaD. melanogaster heads were collected and lysed with strong lysis buffer (CWBIO China Cat# CW2333) and protease inhibitor (CWBIO China Cat# CW2200). Whole tissue lysates were subjected to SDS-PAGE and immunoblotting as described (REF). The molecular weights of 14-3-3ε, Ubx and β-tubulin protein are 30KDa, 40KDa and 50KDa, respectively. The used primary antibodies were as follows: guinea pig anti-14-3-3ε , mouse anti-Ubx , and mouse anti-β-tubulin . The used secondary antibodies were as follows: HRP Goat anti-Guinea Pig IgG , HRP Goat anti-Mouse IgG . Co-immunoprecipitation was conducted as previously described [1118w strain fly heads were collected and fixed by shaking in 1% formaldehyde for 10 min at RT, and cross-linking reactions were stopped by adding glycine at a final concentration of 125 mM at RT for 5 min. The cross-linked chromatin was cut by sonication to approximately 200–500 bp fragments. A 120 μL sample of protein was used for immunoprecipitation, and 10 μL was maintained as the input DNA. The chromatin immunoprecipitation reaction was performed with 20 μL of antibody (mouse anti-Ubx). Immunoprecipitated DNA was quantified by real-time PCR. The ChIP-qPCR data were normalized by the input DNA, and the results were presented as the enrichment fold DNA. The sequences of the primers are shown in Chromatin immunoprecipitation was conducted as previously described . w1118 sAxonal cross was used to quantify the morphology complexity. The data analysis is described in Fernández et al. (2008) . Six eve1118w) and JrkClk mutant flies were sampled at ZT2, ZT8, ZT14 and ZT20 and analyzed using the iTRAQ-MS method. Ingenuity Pathway Analysis (IPA) was used for protein screening. For the detailed methods of experiment and data analysis of proteomic screen, refer to our previous publication [EP357814-3-3ε and 1118w at ZT2 and ZT14 was used for RNA-seq. The RNA-seq was completed by Beijing Biomics Biotech Co. Ltd. . Each sample contained 70 individual flies.Total proteins from whole heads in both wild-type   B. Thus, 14-3-3ε mutant by transcriptomic analysis (p < 0.0001) at ZT2 and 55% (p < 0.0001) at ZT14 in the 14-3-3ε mutant flies (EP357814-3-3ε) compared to those in control flies  compared to that of the control, but it was significantly increased by 89%  at ZT14 (14-3-3ε protein trap fly line (G0008214-3-3ε) fusing 14-3-3ε with GFP (14-3-3ε-GFP) to co-locate the 14-3-3ε and PDF (green for 14-3-3ε by rabbit anti-GFP and red for PDF by mouse anti-PDF). The results showed that both 14-3-3ε and PDF co-expressed in the sLNvs (The pigment dispersing factor (PDF), expressed in the LNvs of clock neurons, is a regulatory factor of sleep in Drosophila. We found that the PDF transcription level decreased in the analysis , which wol flies A. Then,  at ZT14 B,C. Furthe sLNvs A–C. 14-3-3ε is expressed in the PDF neurons, we specifically downregulated its expression with an RNAi driven by pdf-gal4. The results showed that downregulation of 14-3-3ε recapitulated the sleep phenotype caused by the 14-3-3ε mutant, with significant decreases in the total sleep at daytime compared to that of controls  (14-3-3ε regulates sleep via PDF in the PDF neurons.Because ctively) D. These 14-3-3ε regulates PDF, we predicted the transcription factors in the promotor of PDF using the website http://gene-regulation.com/index2.html (accessed on 1 October 2020), in which the Ultrabithorax (Ubx) is one of the transcription factors (Ubx was downregulated in the Ubx heterozygous mutant (1Ubx/+), the pdf mRNA level significantly decreased by 41% compared to that in the control (p < 0.0001)  (To further determine how  factors A. When U 0.0001) B. From t 0.0001) , the Ubx 0.0001) C. Ubx, we designed 10 pairs of PCR primers spanning 2 kb upstream of the translational start sites of PDF (each fragment of which was around 200 bp) for analysis of DNA from ChIP (chromatin immunoprecipitation) with anti-Ubx. The results showed that there was an active peak in fragment 8   E. 14-3-3ε and Ubx, we co-localized Ubx and 14-3-3ε by using the 14-3-3ε-GFP/+; Ubx-gal4/+ fly lines. The brains were stained with immunofluorescence using anti-GFP and anti-Ubx antibodies. The results showed that 14-3-3ε and Ubx were merged together in PDF neurons (EP357814-3-3 by Western blotting. The evidence showed that Ubx significantly decreased by 88% when compared to control (p = 0.004)  E. All th14-3-3ε on sleep regulation, we performed RNA-seq of the head tissue at ZT2 and ZT14 in both the 14-3-3ε deficient mutant (EP357814-3-3ε) and 1118w control flies  or 14-3-3ε and Oamb (M111578/14-3-3εj2B10Oamb). The results showed that decreases in sleep phenotype caused by j2B1014-3-3ε/+ could be partially recovered in these double-mutant flies  or 14-3-3ε and Oamb (M111578/14-3-3εEP3578Oamb). The results showed that decreases in PDF level caused by 14-3-3ε mutant flies could be partially recovered in these double-mutant flies , a neuropeptide secreted from the LNv neurons of the brain, is a wake-promoting factor. When flies are stimulated by light, LNvs respond to light and promote arousal by releasing PDF ,29. Loss14-3-3ε. Ultrabithorax (Ubx) encodes a homeodomain transcription factor involved in cell fate decisions, cell proliferation, and organ identity, and it belongs to the Hox gene family. Hox genes, including Sex-combs reduced (Scr), Antennapedia (Antp), Ultrabithorax (Ubx) and abdominal-A (abd-A), play a conserved role in establishing the thoracic and abdominal segments during insect embryogenesis . Singh e14-3-3ε in sleep regulation. 14-3-3ε, controlled by Clock, regulates both PDF and metabolic factors that are important for neurotransmitter biogenesis. Previous studies showed that multiple types of neurotransmitters had been identified, including acetylcholine (Ach), noradrenaline (NA), histamine, 5-hydroxytryptophan (5-HT), dopamine (DA), glutamate (Glu), and γ-aminobutyric acid (GABA) [14-3-3ε mutant, among which dopamine and octopamine have been proved to be sleep regulators by previous studies in Drosophila. Thus, 14-3-3ε was found to be a novel regulator of neurotransmitters in this study, in which the mutants from the Dop1R1 and Oamb can restore the 14-3-3ε phenotype to different degrees. In this study, a new sleep regulation pathway, i.e., 14-3-3ε, that regulates sleep through the dopamine and octopamine signal pathway, was identified.In this study, we identified the role of d (GABA) . The pro14-3-3 proteins are found to be important in both cancer- and age-related neurodegenerative disease [yki [14-3-3ε is involved in gastric cancer and colorectal cancer [ disease , which case [yki , an impoase [yki ,40. Prevl cancer ,44,45,46l cancer . Many cul cancer ,50,51,52l cancer . HIF-1αol cancer ,49,50,51l cancer .14-3-3ε, controlled by Clock, regulates sleep through pathways of both the 14-3-3ε/Ubx/PDF and neurotransmitters. In addition, there are previous reports that 14-3-3ε is also related to cancers [14-3-3ε is a multifunctional gene in Drosophila regulating different physiological activities. Currently, the direct relationship between cancer and sleep is still unclear, which need to be carefully designed for systematical investigation in future. As is well known, the clock genes regulate the circadian rhythm or/and sleep . In this cancers ,44,45,46


As the demand for rehabilitation in orthopaedics increases, so too has the development in advanced rehabilitation technology. However, to date, there are no review papers outlining the broad scope of advanced rehabilitation technology used within the orthopaedic population. The aim of this study is to identify, describe and summarise the evidence for efficacy for all advanced rehabilitation technologies applicable to orthopaedic practice.The relevant literature describing the use of advanced rehabilitation technology in orthopaedics was identified from appropriate electronic databases (PubMed and EMBASE) and a narrative review undertaken.Advanced rehabilitation technologies were classified into two groups: hospital-based and home-based rehabilitation. In the hospital-based technology group, we describe the use of continuous passive motion and robotic devices  and their effect on improving clinical outcomes. We also report on the use of electromagnetic sensor technology for measuring kinematics of upper and lower limbs during rehabilitation. In the home-based technology group, we describe the use of inertial sensors, smartphones, software applications and commercial game hardware that are relatively inexpensive, user-friendly and widely available. We outline the evidence for videoconferencing for promoting knowledge and motivation for rehabilitation as well as the emerging role of virtual reality.The use of advanced rehabilitation technology in orthopaedics is promising and evidence for its efficacy is generally supportive. With an ageing population, the number of patients affected by orthopaedic conditions that either require rehabilitation alone or surgical intervention and subsequent rehabilitation are on the rise , 2. As tSome advanced rehabilitation technology is bulky, costly and complex and mainly useful for rehabilitation in hospital. However, rehabilitation after many surgical interventions takes place in a patient’s own home . AdherenAlthough there has been growing interest in the use of advanced rehabilitation technologies in orthopaedics, an initial scoping review of the literature failed to identify a paper offering a broad overview of this technology used within an orthopaedic population. Therefore, the purpose of this narrative review is to explore the broad variety of technologies that are currently being used in orthopaedic rehabilitation and determine the extent to which these technologies can support and complement traditional services such as physiotherapy.A literature search of journal articles using the PubMed (MEDLINE) and EMBASE databases was conducted in September 2019. No date restrictions were placed. Relevant literature describing rehabilitation technology utilized in orthopaedics was identified from the above appropriate clinical databases and a narrative review was undertaken.This review paper considers advanced rehabilitation technology into two sections: hospital-based rehabilitation and home-based rehabilitation  and has been primarily advocated to improve knee flexion recovery , 12. NayThe electromagnetic tracking system (ETS) is a six degrees-of-freedom measuring device that simultaneously records the three-dimensional 3D) position and orientation of multiple electromagnetic sensors attached to body segments using a transmitter D positio, 22. TheRehabilitation in patients with spinal cord injury (SCI), who have reduced or absent sensorial input and motor output, is challenging . ParticuRecently, portable robots such as the ARGO , EKSO , Indego , ReWalk  and WPAL  have also been developed, and can perform gait training not only indoors but also outdoors. Hybrid Assistive Limb  has a hybrid system allowing both a voluntary and an autonomous mode of action to support gait training. HAL uses control algorithms and supporting devices that control each knee and hip separately. A systematic review reported that although HAL had beneficial effects on gait, function and independence in walking, no study provided conclusive data on differences between HAL gait-training compared to the other forms of gait-training . On the The effect of robotic devices on upper limb rehabilitation in SCI patients is less established. Zarffa et al. reported no difference in the improvement in functional scores for patients receiving Armeo Spring®  training on one arm compared to the arm that did not have robotic-assisted rehabilitation, both at discharge and follow-up assessment . To dateKoller-Hodac et al. introduced a robot-assisted knee device that can be used at home to acquire a full ROM and joint flexibility after knee surgery . A narraRobotic devices face a number of challenges including the pressure of the suit in certain areas, skin irritation, training-related pain, excessive energy consumption due to device use and high costs , 33. UsiInertial sensors such as accelerometers, magnetometers and gyroscopes collect 3D motion data, which they communicate to a computer for analysis by accompanying software. This allows them to be used to accurately measure and assess the movement of a patient’s joint during a variety of functional exercises . Saber-SThere are many medical Apps available, with some offering knowledge of medical conditions, anatomy, drug information or other treatments . The relTM  for elastic band exercises. It can quantify contraction time, the number of repetitions performed and the force used to stretch the elastic band by measuring the thickness of the band [TM can send data to an iPad, and the BandCizerTM App provides users with real-time feedback on exercises [TM-iPad feedback on contraction time or not. The App significantly improved compliance to the rehabilitation [TM application could be utilised across multiple areas of orthopaedic physiotherapy rehabilitation [Apps are also useful for patient feedback. One technology that provides real-time feedback is a sensor called BandCizerthe band . BandCizxercises , 54. Ratlitation . As elaslitation .However, Stove et al. emphasised that subtle variations exist in the quality of measurements depending on the smartphone manufacturer and also that updates of application software complicate the validity of the findings and may lead to inconsistencies in the assessment of measurement such as ROM . TherefoVideoconferencing (VC) has the potential to play an important role in the management of orthopaedic patients. VC could allow therapists to diagnose, treat and monitor patients’ progress remotely as telerehabilitation in orthopaedic setting.Eriksson et al. investigated patients’ experiences of participating in therapy via VC at home for 2 months after shoulder joint replacement . This stTousignant et al. reported on satisfaction with VC for patients following a RCT post TKR . AlthougVC promotes the relationship between patients and therapists, knowledge and motivation for rehabilitation, and may reduce costs . HoweverCommercially available gaming systems (CAGS) can detect subtle changes in balance, coordination, strength and ROM of a joint during functional activities . CAGS maMany studies have shown Microsoft Xbox with Kinect to have an excellent correlation and reliability when compared to a gold-standard motion detection system such as the Vicon MZ motion analysis system  , 64. HowYamada et al. reported that the score of Wii Fit  and Basic Step game correlated with Dual Task Lag of Timed Up-and-Go in older women, which demonstrated association with a real-life outcome . A RCT sCAGS have the potential to make an important contribution in orthopaedic rehabilitation but their role in this context is still in development. Whilst most studies are favourable, some studies have not shown the effectiveness of CAGS when compared to the gold standard . Strong Many advanced rehabilitation technologies relevant to orthopaedics are still in development. Virtual reality (VR) technology, which includes an interactive computer environment or games that appear and feel real may also have a role. In physical rehabilitation, VR can be used to personalise treatment, motivate patients, improve compliance and track progress. Currently, there is a lack of strong evidence to support its use. Clinical trials have assessed VR effectiveness in patients with orthopaedic pathology such as ACL injury, frozen shoulder and chronic neck pain. Most of them used off-the-shelf console games such as Nintendo Wii Fit, making it difficult to differentiate from rehabilitation using CAGS .Electromyography (EMG) muscle stimulation devices have shown significant improvements in outcome for patients after TKR . CurrentTo date, a number of protocol papers for advance rehabilitation technology RCTs have been published in the orthopaedic field –75. As tThis narrative literature review describes the variety of advanced rehabilitation technologies that are currently used in orthopaedics. It outlines the evidence for the extent to which these technologies can support and complement traditional therapy.Hospital-based technology, such as robotic devices, is widely used mainly in SCI, and it is expected that they will be utilised for other conditions in the future. The reliability of ETS for measuring kinematics of upper and lower limbs has been reported, but the challenge is to develop compact and user-friendly devices. Unlike hospital-based advanced rehabilitation technology, home-based technologies such as inertial sensor, application, and CAGC are relatively inexpensive and user-friendly, making them more accessible. A number of these modalities have been shown to be effective measurement tools in orthopaedics through accurate quantification of patient physical activities such as dynamic ROM and function. VC, a method of telerehabilitation, is acceptable to most patients, and it promotes the patient-therapist relationship and the patient’s knowledge and motivation for rehabilitation.The use of advanced rehabilitation technology in orthopaedics shows a lot of promise, particularly to support and complement traditional rehabilitation services, and its use is gaining in popularity. These technologies are dependent on the device accuracy and reliability. There remains a paucity of high-level published evidence as to efficacy. Further research is needed to determine the usability, cost-effectiveness and efficacy of advanced rehabilitation technology in high-quality randomised cohorts of orthopaedic patients.


In the presence of Cs+ cations, the Ag29(SSR)12 nano-building blocks are selectively assembled into distinct arrays with different oxygen-carrying solvent molecules―Cs@Ag29(SSR)12(DMF)x as 1D linear chains (29–1DAg), Cs@Ag29(SSR)12(NMP)x as 2D grid networks (29–2DAg), and Cs@Ag29(SSR)12(TMS)x as 3D superstructures (29–3DAg). Such self-assemblies of these Ag29(SSR)12 units have not only been observed in their crystalline state, but also in their amorphous state. Due to the diverse surface structures and crystalline packing modes, these Ag29-based assemblies manifest distinguishable optical absorptions and emissions in both solutions and crystallized films. Furthermore, the surface areas of the nanocluster crystals are evaluated, the maximum value of which occurs when the cluster nano-building blocks are assembled into 2D arrays (i.e. 29–2DAg). Overall, this work presents an exciting example of the hierarchical assembly of atomically precise nanoclusters by simply controlling the adsorbed molecules on the cluster surface.The supramolecular chemistry of nanoclusters is a flourishing area of nano-research; however, the controllable assembly of cluster nano-building blocks in different arrays remains challenging. In this work, we report the hierarchical structural complexity of atomically precise nanoclusters in micrometric linear chains (1D array), grid networks (2D array) and superstructures (3D array). In the crystal lattice, the Ag 29(SSR)12 nano-building blocks with different solvent-conjoining Cs+ cations.Micrometric linear chains (1D array), grid networks (2D array), and superstructures (3D array) of nanoclusters were selectively constructed by hierarchical self-assembly of Ag More specifically, each outward Cs+ cation connects one cluster1 and one cluster2, of which the cluster1 is arranged downwardly but the cluster2 is organized upwardly, constructing the Ag29-based, three-dimensional superstructures . In this context, the cluster packing modes are identical in all directions, making up a highly symmetrical superstructure 12 triggers the self-assembly of cluster dots into superstructures, representing the three-dimensional arrangement of the Ag29 nano-building blocks in the crystalline cell.For the 3D-array assembly  exhibited distinct crystal structures and crystalline packing modes . Although the overall Ag29(SSR)12 configuration retained from 29–0DAg to 29–1DAg,Due to the different surfaces, these Ag29–2DAg and 29–3DAg, obvious changes have been observed by comparing the corresponding bond lengths. Specifically, all of the three types of Ag-Ag interactions (Ag(core)–Ag, Ag–Ag, and prism-like Ag–Ag(motif) bonds) in Cs@Ag29 nanoclusters  were much longer than those in the PPh3@Ag29 nanocluster (29–0DAg), demonstrating an expanding trend of the overall framework along with the PPh3 dissociated process . For the pyramid-like interactions between the vertex Ag and the icosahedral Ag, the bond lengths were all close to 3.04 Å for Cs@Ag29 nanoclusters. However, no analogous interaction was observed in the PPh3@Ag29 nanocluster since the corresponding distances ranged from 3.493 to 3.643 Å . In this context, the vertex Ag atoms became closer to the icosahedral kernel when the 29–0DAg nanocluster was transformed into 29–1DAg, 29–2DAg and 29–3DAg, and the newly generated Ag4 pyramids were anticipated to make the Ag29(SSR)12 framework more robust.+ ions in different Ag29-based assemblies have been compared. For 29–1DAg, three Cs+ ions  stabilize the cluster surface and the other three Cs+ ions  assemble Ag29 nano-building blocks into 1D linear chains  stabilize the cluster surface, and the outward Cs+ ions  induce the inter-cluster assembly of cluster nano-building blocks into 3D superstructures . Of note, two types of crystallization patterns of 29–0DAg have been reported―29–0D-cubicAg and 29–0D-trigonalAg―due to their different crystallization processes 3−, [Ag29(SSR)12(PPh3)3]3−, [Ag29(SSR)12(PPh3)2]3−, [Ag29(SSR)12(PPh3)1]3− and [Ag29(SSR)12]3−, respectively, in good agreement with the reported ‘dissociation-aggregation pattern’ of the PPh3 ligands on the 29–0DAg surface  3− and [CsAg29(SSR)12]2− compounds, were observed for each mass spectrum of 29–1DAg, 29–2DAg or 29–3DAg , which verified the Cs+ capture in these nanoclusters. However, as to the mass spectra of each Cs@Ag29 nanocluster, only the single Cs+-adhered Ag29 compound (i.e. CsAg29(SSR)12) could be detected. The unattained mass signals of the complete Cs@Ag29 molecules resulted from the weak interactions among the Ag29(SSR)12 frameworks, the Cs+ cations, and the solvent molecules when the nanoclusters were in solutions.The electrospray ionization mass spectrometry (ESI-MS) measurement was firstly performed to verify the specific composition of each nanocluster . Mass spectra of ig. S7A) . These P133Cs and 31P nuclear magnetic resonance (NMR) were then recorded to validate the capture of PPh3 ligands or Cs+ ions on the Ag29 nanocluster surface. As depicted in Fig. S8A, the 133Cs NMR of CH3COOCs showed an intense signal at 72.74 ppm, and this signal shifted to high fields  when the Cs+ cations were captured by the Ag29(SSR)12 framework. The different 133Cs NMR signals of Cs@Ag29 nanoclusters originated in the distinct cluster–Cs–solvents interactions of these Ag29-based assemblies. Besides, the intense 31P NMR signal of 29–0DAg at 26.20 ppm disappeared after the PPh3 ligands were dissociated from the nanocluster surface ; that is, no phosphine signal was observed in the 31P NMR of Cs@Ag29 nanoclusters.29-based assemblies manifested distinguishable optical absorptions and emissions in both solutions and crystallized films. Of note, the solution-state UV-vis and photoluminescence (PL) spectra of 29–0DAg and 29–1DAg were monitored in DMF, whereas 29–2DAg was in NMP and 29–3DAg was in TMS. The optical absorptions of these Ag29 nanoclusters in the solution state were very similar 12 framework. For the PL, all nanocluster solutions emitted when illuminated at 445 nm  and 29–3DAg (in TMS) exhibited obvious blue-shifts, of which 29–2DAg emitted at 625 nm and 29–3DAg luminesced at 622 nm. Furthermore, the PLThe structures of nanoclusters are determinant of their physical-chemical properties. Due to their distinct surface structures and crystalline packing modes, these Ag29–0DAg, 29–1DAg and 29–3DAg showed 1.7-, 2.1- and 2.3-fold enhancement, respectively, relative to that of 29–2DAg with the lowest PL intensity. These differences reflected both the structural effect and the solvent effect on nanocluster emissions.intensities of 29–2DAg was much more intense than those of other nanoclusters, and the UV-vis spectrum of 29–1DAg showed a broad shoulder band at 550 nm that was absent for other nanoclusters  of 29–3DAg was more distinguishable than that of the 29–2DAg  arose from distinct combinations of the electronic coupling and the lattice-origin, non-radiative decay pathways occurring through electron–phonon interactions [3 ligand surface of 29–0DAg, and the distinct cluster–Cs–solvents surfaces of 29–1DAg, 29–2DAg and 29–3DAg.The nanocluster crystallized films exhibited apparent differences in both optical absorptions and emissions Fig. B. The UVractions ,52,53. B29-based assemblies should exhibit distinctive surface areas. Herein, the nitrogen adsorption–desorption tests were performed on the crystals of these Ag29 nanoclusters for evaluating their specific surface area and pore size distribution . By comparison, the 29–2DAg crystal generated a much bigger specific surface area of about 19 m2/g. In this context, as to this Ag29 system, the nanocluster crystals would expose the maximum surface areas when the cluster nano-building blocks were assembled into 2D arrays. Indeed, compared with other cluster crystals, the 29–2DAg crystal presented larger pore sizes 12 nano-building blocks with different solvent-conjoining Cs+ cations. In the absence of Cs+ cations, the bare Ag atoms on Ag29(SSR)12 were prone to be stabilized by PPh3 ligands, producing the unassembled cluster dots in the crystal lattice. In the presence of Cs+ cations, the Ag29(SSR)12 units could be selectively assembled into distinct arrays with different oxygen-carrying solvents: Cs@Ag29(SSR)12(DMF)x as 1D linear chains with the DMF solvent, Cs@Ag29(SSR)12(NMP)x as 2D grid networks with the NMP solvent, and Cs@Ag29(SSR)12(TMS)x as 3D superstructures with the TMS solvent. Besides, the 1D–3D self-assemblies of these Ag29(SSR)12 nano-building blocks have not only been observed in their crystalline state, but also in their amorphous state, with the help of the aberration-corrected HAADF-STEM. Such Ag29-based assemblies manifested distinguishable optical absorptions and emissions in both solutions and crystallized films, and these differences originated from their different surface structures and crystalline packing modes. The surface areas of these Ag29 crystals were evaluated, and the 2D-array assembled nanocluster (i.e. Ag29-based grid networks) displayed the maximum value of the surface area. Overall, this work presents the hierarchical assembly of atomically precise nanoclusters by simply controlling the adsorbed molecules on the cluster surface, which hopefully sheds light on more future works touching upon the supramolecular chemistry of metal nanoclusters.The cluster-based 1D linear chains, 2D grid networks and 3D superstructures were selectively constructed by the self-assembly of Ag3, 99%, metal basis), triphenylphosphine , 1,3-benzene dithiol , sodium borohydride , cesium acetate , methylene chloride , methanol , N, N-dimethylformamide , N-methyl-2-pyrrolidone , tetramethylene sulfone , and ethyl ether ((C2H5)2O, HPLC, Sigma-Aldrich).All reagents were purchased from Sigma-Aldrich and used without further purification: silver nitrate . This NMP solution of 29–2DAg was directly used for the crystallization and the characterization.The 50-mg 29–1DAg crystal was dissolved in 5 mL of TMS under vigorous stirring. This TMS solution was poured into 200 mL of CH2Cl2, and the precipitate was collected and further dissolved in 5 mL of TMS, producing the 29–3DAg nanocluster. The yield was 95% based on the Ag element . This TMS solution of 29–3DAg was directly used for the crystallization and the characterization.The 50-mg 29–0DAg and 29–1DAg were cultivated based on the reported methods [29–2DAg and 29–3DAg were cultivated at room temperature by diffusing methanol into the NMP solution of 29–2DAg, or the TMS solution of 29–3DAg. After two weeks, red crystals were collected, and the structure of 29–2DAg or 29–3DAg was determined. The CCDC numbers of 29–2DAg and 29–3DAg are 1961389 and 1941329, respectively.Single crystals of  methods ,49. Singv/v = 1:2) by methanol. 133Cs and 31P NMR spectra were acquired using a Bruker 600 Avance III spectrometer equipped with a Bruker BBO multinuclear probe . The Ag29-based assemblies were imaged with an aberration-corrected HAADF-STEM technique after the solvent that contained Ag29-based assemblies was dropped casting onto ultrathin carbon film TEM grids. The microscope employed was a FEI Themis Z. The electron beam energy was 200 kV. The collecting angle HAADF detector was used to collect signals scattered between 52 (inner angle) and 200 (outer angle) mrad (camera length of 146 mm). The aberration-corrected HAADF-STEM image was obtained by Thermo Scientific Velox software using 1024*1024 pixels and dwell time was set to 10 us.All UV-vis absorption spectra of nanoclusters were recorded using an Agilent 8453 diode array spectrometer. PL spectra were measured on a FL-4500 spectrofluorometer with the same optical density of 0.1. ESI-MS measurements were performed by MicrOTOF-QIII highresolution mass spectrometer. The sample was directly infused into the chamber at 5 μL/min. For preparing the ESI samples, nanoclusters were dissolved in DMF/NMP/TMS (1 mg/mL) and diluted  is remarkably higher than those of the 29–0DAg, 20–1DAg and 29–3DAg samples.The specific surface area and pore size distribution were calculated from each corresponding nitrogen adsorption–desorption isotherm by applying the Brunauer-Emmett-Teller (BET) equation on ASAP2020 M plus Physisorption. By using the quenched solid density functional theory (QSDFT), the pore size distributions were derived from the sorption data. The BET surface areas of 2 using the SHELXTL software package. All non-hydrogen atoms were refined anisotropically, and all the hydrogen atoms were set in geometrically calculated positions and refined isotropically using a riding model.The data collection for single crystal X-ray diffraction was carried out on Stoe Stadivari diffractometer under nitrogen flow, using graphite-monochromatized Cu Kα radiation (λ = 1.54186 Å). Data reductions and absorption corrections were performed using the SAINT and SADABS programs, respectively. The electron density was squeezed by Platon. The structure was solved by direct methods and refined with full-matrix least squares on Fnwaa077_Supplemental_FilesClick here for additional data file.


Spring is a fertile season for new thinking, and I wish to share new ideas. Some discussions I have had with certain medical physicists over the past several days have been especially thought‐provoking and caused me to reflect on some of the character and substance of medical physics practice. I herewith offer some personal anecdotes that resonate with my experience of medical physics education and practice along with some contemporary thoughts.April 9–10th‐grade math class. Dyke Goodin, my instructor, offered that anyone who earned a 100 average on all tests, given weekly, would earn an A+ in the class. I took him up on this challenge as I thought I only had to be careful and I could do it. In 2018, I contacted Dr. Goodin, and asked if he remembered me. I wrote: “One of your challenges to our class, so long ago, was to strive for perfection. Your standard for an A+ was daunting. Nothing less than a 100% average on all weekly tests (with no extra credit) including the final would earn it. I determined I would do it, and indeed, in the final quarter, I earned that elusive and rare A+. Let me tell you what that did for me. I had the confidence I would go as far as I could go in whatever my calling chose for me.” His response: “It might behoove you to know that you were the first student to make a 100 average without bonus points. There have been only 17 young people to accomplish this feat in my 47 years. I remember you very well. You thrived on the challenge of academic excellence. You made a 100 on my most challenging test.” My 50th (plus 1) class reunion is in June, and I hope to see Dr. Goodin in Atlanta for the first time in 54 years. In 1966, the goal to strive for perfection was unquestioned. A career in medical physics may mean over 50,000 calculations, all of them perfect or a patient may suffer injury or worse.My first story stems from 1966 and my 8My second anecdote (from 1973) comes from when I was working at Doctor’s Hospital in Tucker, Georgia. A gentleman having a lumbar spine exam was asking a lot of technical questions of our flustered technologist. I was a senior physics major and I volunteered to answer his questions. He asked about the grid, the bucky, the energy, the collimation, the radiation safety procedures in place, and a few other questions; I was able to satisfy him. He asked me what I planned to do, and I told him I was planning to attend the MS program in Health Physics at Georgia Tech next year. He smiled and said he would see me in class. I looked up and remembered his name: Karl Z Morgan. Some of you may recognize who he was: The Director of Health Physics at the Oak Ridge National Laboratories in the 1940s and 1950s, the founder of the Health Physics Society and the first Editor of the Health Physics Journal. He became my advisor for my first Master’s degree.My third story is from my PhD Advisory Committee meeting at the MD Anderson Cancer Center. My advisor was the renowned Ray Meyn, one of the world’s leading radiobiologists and ultimately the acting Chair of the Department of Experimental Radiotherapy at the MD Anderson Cancer Center. However, in 1977, he was an Assistant Professor just beginning his career and I was his student. I put together a project, which I thought was acceptable. His friend and a member of my committee was a very bright scientist whose name was Roger H_. Every student was afraid of him. After presenting my project, Dr. H_ said, “all of this is good, but what is your goal in accomplishing this project since you want to become a medical physicist?” I replied, “my goal is to do such a good job with the experiments, the writing, the conclusions and the publication of this work that it makes my supervisor, Dr. Meyn, an eminent scholar, known around the world for the value of his creative science.” Ray Meyn chipped in, “Well, now that is a goal and ambition I can wholeheartedly endorse!” Ultimately, the two articles from this project received 156 citations, according to the Web of Science. Ray told me in 2007 that this was the most cited project in his career.My final anecdote stems from when I was teaching a laboratory in radiation safety in 1979. It was about 7:30PM and the clinic was empty. I had two students, and we had access to the Atomic Energy of Canada Limited (AECL) Therac 20. I set the energy for 18 MV, the jaws for 40 x 40 cm, pointed the unit at 180 degrees (straight up), and told the students to find where the beam was coming out and measure it with the survey meters I provided. After one minute, I activated the beam. After 10 minutes, they returned. The beam was exiting into a physician’s office with the center of the beam right under the chair! The office was occupied by Dr. Bill Spanos, the man who ultimately became Chair of Radiation Oncology at the University of Louisville, and the gentleman who hired me to be Chief of Physics. His message through my students was: “Thank you very much, I appreciate knowing about this.” I later found out that, at least for some of his patients, four field techniques were eliminated in favor of three field wedged techniques, removing the 180‐degree beam.So many in our culture approach life seeking to obtain power and to put themselves in a position where they do not need to practice excellence and are free from being questioned or criticized about their true goals and ambitions. Today, we can also be faced with significant social criticism of such perfectionistic goals and ambitions in education as a challenge to attain a 100 average. Does our culture welcome critical questions, or are we more often expected to trust powerful authorities without questioning them? Is it all about us and our goals, or is it still about making our employers and supervisors successful? Do we relentlessly strive to eliminate even small risks or do we minimize some significant risks while greatly magnifying others for personal gain or convenience? Is it wise to hold such self‐interested thinking and to welcome and empower those with such attitudes in the medical physics profession? How will this affect the practice of medical physics and ultimately the outcome of our patients?What things do we not know that are putting ourselves, our patients or our employers at risk? After striving for excellence, asking critical questions, focusing on our goals, and examining the limits of our knowledge in a profession we are spending a lifetime to master, we often still find ourselves quite vulnerable to the unknown. Patients are ultimately best served by medical physicists who strive relentlessly for excellence, in a work environment unafraid to identify deficiencies or errors. Medical physics differs from many contemporary vocations with flexible missions where the time’s social priorities merge naturally with the profession’s flexible mission. At the end of the day, all medical physics work critically and heroically deals with human lives, their conditions improved or harmed by our actions according to the mindset we embrace.


We present a data-driven model that rates actions of the player in soccer with respect to their contribution to ball possession phases. This study approach consists of two interconnected parts: (i) a trajectory prediction model that is learned from real tracking data and predicts movements of players and (ii) a prediction model for the outcome of a ball possession phase. Interactions between players and a ball are captured by a graph recurrent neural network (GRNN) and we show empirically that the network reliably predicts both, player trajectories as well as outcomes of ball possession phases. We derive a set of aggregated performance indicators to compare players with respect to. to their contribution to the success of their team. Analyzing team and player behavior in soccer games is an inherently difficult task: coordination among teammates is highly complex and even well-developed game plans can be negated by a single bad pass. In these circumstances, the contribution of a single player to the success of a team is often difficult to estimate. While obvious decisive actions, such as scoring and assisting a goal, losing the important tackle just before the scorer takes the shot, can easily be measured and attributed to those playersLast but not least, there is a great deal of randomness in the game of soccer and the outcome highly depends on chance. A good team may still lose the game simply because they hit the post instead of the goal, while their opponents capitalize on an individual error. Still, the good team may have created more scoring opportunities and played better in general and when it comes to player rating, the players of the good team should be rated accordingly.Because of the complexity in which a soccer match develops, there has been a lot of research on specific aspects of the game to analyze and measure their potential influence on the outcome. For example, Spearman et al.  developeIn this article, we devise a model that predicts player trajectories and estimates the expected outcome of a ball possession phase. A ball possession phase is a time between a team gaining possession of the ball and loosing it again. Ultimately, the outcome of such a sequence could either be a success by scoring a goal or a failure by losing possession to the opponent. However, as goals are rather rare in soccer, success can also be defined less precisely, e.g., by the expected goal value in case of a shot or, even more general, by entering a certain area on the pitch that potentially allows for a scoring chance. In this article, we use the latter approach and consider a built-up play, a success if the team enters a dangerous zone while retaining ball possession. The basis for the model is a graph recurrent neural net (GRNN) that models interactions between players and the ball using a fully-connected graph, where all players and ball are represented by nodes. Their interdependencies are given by edges between the nodes and are, finally, learned by the GRNN to predeict the outcome of a ball possession phase. To this extent, the model also learns node features for every node to predict the movement of the corresponding player or ball. This research borrows ideas from Dick and Brefeld  to learnWe learn and evaluate the model on data from 54 Bundesliga matches where ball and player positions are captured at 25 frames per secondThe remainder of the article is structured as follows. We briefly review related work in the next section and present the main contribution in section 3 on empirical results in section 4, and section 5 provides a discussion of practical applications in section 5, and give conclusion in section 6.Rating game states and player actions in soccer is a fairly new topic that aims at quantitatively measuring how particular actions, such as passes or dribbles, influence the outcome of a game or how likely a game configuration consisting of ball and player positions, control of the ball, or recent actions, among others, may lead to a scoring opportunity. Data sources used to ground models on observable data generally consist of event data, such as information about shots and passes and/or trajectory data that measures the position of players and the ball several times a second. Due to the inherent complexity of soccer matches, most work has focused on rather specific aspects of soccer to rate those particular actions, such as passes or shots. Shot opportunities were investigated using the expected goal  value, e.g., in Lucey et al.  with theMore general approaches include a study by Fernández et al.  who deveAnother aspect of soccer analytics that received attention is the simulation of player trajectories. Peralta Alguacil et al.  partly cSeveral studies instead investigated learning to predict player trajectories in soccer and other sports, such as basketball . Several of those shots and goals originated from set pieces and are not suitable for use as an outcome of a ball possession phase. As a consequence, we focus on a proxy that appears more frequently in the data to facilitate learning with more evenly balanced success signals. Thus, instead of defining success simply by goals or shots, we propose to consider ball possession phases “successful” that ends with the ball possessing team entering a agents, that is players and the ball, as nodes in a (fully connected) graph. Interactions and interdependencies between these agents are encoded by edges between pairs of agents. In addition, we include a graph state of the graph which encodes information about the whole graph, cmp. e.g., Battaglia et al.  that each encodes the whole graph and transforms node and state features based on information of the previous time step and input features. I and ρI to create initial node and graph state representations uT0, . Those initial features are passed through multiple recurrent GL to output final node features uT. To provide more details, we begin by presenting the basic steps that are performed in a single GL ℓ.The big picture is as follows: At each time step, input features uTℓ−1,  from the previous layer ℓ−1 and input features Tℓ,  using another attention mechanism which is also fed into a GRU unit to compute transformed state features uTℓ, . Equations (1) to (5) show the detailed computations to compute transformations At first, all edge features h et al. . Each edtype and αv, and function, fv, are implemented with multilayer perceptrons (MLP) with ReLU activations and ⊙ denotes the element-wise product. The initial node and state features, uT0, , that serve as input to the first GL are computed by Equations (6) and (7) from the input features,Attention functions, αI is realized by a two-layer MLP. Overall, the GRNN transforms input features to node and graph state features, i.e., where feature function ϕMovements: The prediction of the movement of player i at time T is based on node features x of i in time t, that is xy-position of player i at time T. MDNs model the output distribution with a GMM with K mixtures is given by the Equation (8)h et al. , the movI is the identity matrix. The model is learned by minimizing the log-likelihood of observed relative movements.The categorical mixture distribution π is modeled using a two-layer MLP with ReLU activations and a standard softmax output. Similarly, Gaussian means Success: Predicting the success during a ball possession phase demands information about all contributing parties and the natural choice is to use the graph state uT to estimate whether the attacking team can successfully conclude the phase. The approach is to use ideas from reinforcement learning and learn a value function (V) that maps states to the expected (discounted) reward of a possession phase. The reward x-direction completes their ball possession, i.e., the ball V is defined as the expected discounted future reward from state T until it ends at some time TE and γ∈ is a discount factor. The effective V range thereby is between –1 and 1. We model the V using the graph state as V is a two-layer MLP with ReLU activations and linear output. The model is learned with the lambda-return algorithm , a reward of -1 if the team plays from right to left, and a zero reward otherwise. Overall, the data consists of 10,626 ball possession phases with an average length of 10.6 s 1,879 of those possession phases have a non-zero success signal. We report averages of 5-fold cross-validations; error bars indicate SE.We extract all episodes of open play where one team continuously retains ball possession without the game being halted. Each such ball possession phase ends with the team controlling the ball either losing the ball or the play is stopped, or with that team performing a “GL, where both layers have a GRU width of 1, 024 and the attention function, fv, possess a single hidden layer with 512 units. Input feature function ϕI has two layers with 128 and 256 units, respectively. The MDN consists of a Gaussian mixture with k = 6 components and GMM functions, π, μk, and σk, each have two layers with 512 units. The same holds for ϕV. The architecture of the GRNN used in the experiments consist of two We focus on the evaluation on three aspects: section 4.1 verifies whether the learned V translate to actual dangerous situations; section 4.2 evaluates the quality of the predicted player trajectories; and section 4.3 shows how both aspects can be combined to allow the assessment of player performance.The V assigns values to situations on the pitch. The higher the absolute value of the V, the more likely a team should be able to successfully finish a ball possession phase.Trivially, the potential for a successful ball possession highly depends on the ball position: the closer to the goal of the opponent's, the higher the likelihood of success. This can be verified in independent of ball position, we compute AUCs for small areas of the pitch individually. That way, ball position is roughly the same inside an area and differences in V stem only from ball and player positions and trajectories of the recent past. The results are shown in We test for both demands by relating the V to successful outcomes of ball possession phases by measuring the area under the ROC curve (AUC). To evaluate for predictability of success The figure displays higher AUC values for areas closer to the goal. An intuitive explanation is that, being so close to the dangerous zone, chance has less effect on the outcome of the ball possession phase as only a few more actions are required to successfully transfer the ball into that zone. Interestingly, the AUC never falls below 0.64, even well inside the own half of the attacking team (center-line is at 52.5 m). This may at first be considered a surprisingly good result, given that soccer is a highly unpredictable game where little things can change the outcome of a ball possession phase significantly. However, studying the results more carefully discloses that counterattacks, in general have high values accompanied by high success rates and often start deep inside the own half of the attacking team, which partly explains those good AUC values. ConvNet originally presented in Dick and Brefeld (x-coordinate), in contrast to the evaluation in y-coordinate of the ball position was also considered. This evaluation is in line with the one presented in Dick and Brefeld  and feeding those new positions back into the GRNN.xy positions of predicted trajectories and observed trajectories. We empirically quantify deviations of simulations vs. ground truth by measuring the mean distance of attacker show the deviations of players of that category. Every curve shows averages and SE over several players but only one player is simulated at a time. This way, the differences in playing positions become obvious. Defenders need to be synchronized and their movement predictions are close to their real positions. Interestingly, simulating all defending players simultaneously, shown by the curve labeled backline, has the same deviation as the one for only a single defender. Forward players like wingers and strikers are the most difficult ones to predict, which is in line with the findings of Le et al.  led to him being available for the pass.In analogy to the previous tables, For many other circumstances, assigning a value to actions of thee player are less obvious and alternative approaches have to be used. One way to estimate a contribution of the player toward the potential for success of a game situation is by evaluating positioning of the player on the pitch, something which is often done during televised match analyses. The idea is to analyze how a decisive action, such as a forward pass, would have developed, or if it would have happened at all, if a certain player, often a defender, would have been positioned otherwise. We show that this model can be used to estimate quality of individual player movements and positioning and thereby quantify a influence of the player on the V.xy coordinate of that player for the specific frame, and use the new V prediction of the model to estimate whether the new positioning would potentially affect the outcome of the ball possession phase. However, as the deep graph model employs recurrent GRU units, simply changing a position of the player could change the outcome of the prediction in undesired ways because the “simulated" trajectory would be physically impossible, e.g., by changing the position between two frames (0.04 s) by 2 m. Manually simulating trajectories of the player are possible but can be tedious.The simplest way to use this model in that way would be to change the position of the player in the data artificially, i.e., change the Instead, we propose to evaluate off-ball actions of players by comparing their movements to movements of the average (Bundesliga) player that is naturally approximated by this model. As described above, this model is able to predict player trajectories realistically and can be used to simulate how a player could have acted instead. Comparing the temporal development of value functions between observed player trajectories and those where a trajectory of the player is replaced by the simulated one can be used to evaluate the (relative) quality of the observed actions.−4), which again shows that simulated players behave very similar to observed ones; however, while during non-successful ball possessions—i.e., successful from the point of view of the defender, the average values of simulated players are more or less indistinguishable from observed one, simulations are on average a bit better during successful ball possession phases, unsuccessful from the point of view of the defender. In other words, during successful attack, players of the defend team sometimes behave worse than the average player and thereby increase the likelihood of the other team running a successful attack. Interestingly, according to the model, midfielders, wingers, and attackers a had stronger negative influence on the outcome than defenders. The previous section shows that these contributions accurately predict the movements of players as well as the potential for success of a game situation. We also indicate that there are various use-cases, where temporal changes in the V can be credited to a player or team performance; however, there are certainly many more applications where the use of this approach could be beneficial.The previous section also indicates the potential for computation of novel KPIs, such as the conversion rate of dangerous situations into goals and/or shots, attributed to both teams and players. Other performance metrics could measure where most of potentially dangerous attacks of a team originate or in which areas a team is most dangerous. Those metrics are more general than common indicators that measure where successful  attacks originate or develop.The V and derived performance indicators can also be used in retrieval systems to quickly find situations of interest. While trivial situations like corner and free kicks, shots, or passes are already annotated in the event data, complex events, such as a pressing situation or possibilities for counter-attacks are often tedious to identify and require human effort. Particularly for opponent analysis or scouting, this approach may come in handy since queries asking for the dangerous situations in a certain part of the pitch or good actions of a certain player can be processed automatically. As an example, potentially dangerous counter-attacks could be retrieved by querying for high V values inside a own half of a team. On the other hand, querying for situations where a team gains possession of the ball just inside their own half  but low V could return sets of game situations that show how well a team is positioned against counter-attacks.A limitation of this approach is clearly the data. Players are represented only by their coordinates on the pitch while their pose is ignored. Hence, it is not always clear whether players are standing or lying, walking forward or backward, and a player who looses the ball due to a tackle may be pushed to the ground (without being fouled) in the course of the action. In that case, the player is physically not able to track back immediately, a fact that is unknown to the model.I (Equation 6).Due to the conceptual approach, this model provides the same solution for all players in the same situation. It is not personalized in the sense that it remembers players across games or takes history of a player's during a game into account. Thus, it is oblivious to a state of the player regarding mental and physical fatigue. On one hand, personalization would require access to additional data sources, such as for measurements of the mental or physical state of the player including load monitoring, self-perception of fatigue, stress, hours of sleep, muscle pain; however, the incorporation of such “orthogonal” data also opens up new possibilities, since the model could learn about individually preferred movements and physical and mental strength. Such additional information could easily be integrated into the model by either incorporating a player embedding layer for personalization or by adding additional physiological data into the input function ϕWe presented a data-driven model for predicting player trajectories and estimating the potential success of a game situation in soccer. The model employs a GRNN that effectively captures relations about all players and the ball and enabled us to make predictions based on the state of the game. Empirically, we observed that the model learned realistic movements of players and significantly outperformed a baseline approach and that learned V are a good predictor of the outcome of ball possession phases. We derived novel performance indicators by inspecting temporal developments of the V and showed that those correlated well with human expertise. In addition, by combining trajectory and success prediction, we were able to identify the sub-optimal behavior of players.The data analyzed in this study is subject to the following licenses/restrictions: The data is owned by DFL and must not be shared with 3rd parties. Requests to access these datasets should be directed to www.dfl.de.All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Augmented reality (AR) offers a new medical treatment approach. We aimed to evaluate frameless (mask) fixation navigation using a 3D-printed patient model with fixed-AR technology for gamma knife radiosurgery (GKRS). Fixed-AR navigation was developed using the inside-out method with visual inertial odometry algorithms, and the flexible Quick Response marker was created for object-feature recognition. Virtual 3D-patient models for AR-rendering were created via 3D-scanning utilizing TrueDepth and cone-beam computed tomography (CBCT) to generate a new GammaKnife Icon™ model. A 3D-printed patient model included fiducial markers, and virtual 3D-patient models were used to validate registration accuracy. Registration accuracy between initial frameless fixation and re-fixation navigated fixed-AR was validated through visualization and quantitative method. The quantitative method was validated through set-up errors, fiducial marker coordinates, and high-definition motion management (HDMM) values. A 3D-printed model and virtual models were correctly overlapped under frameless fixation. Virtual models from both 3D-scanning and CBCT were enough to tolerate the navigated frameless re-fixation. Although the CBCT virtual model consistently delivered more accurate results, 3D-scanning was sufficient. Frameless re-fixation accuracy navigated in virtual models had mean set-up errors within 1 mm and 1.5° in all axes. Mean fiducial marker differences from coordinates in virtual models were within 2.5 mm in all axes, and mean 3D errors were within 3 mm. Mean HDMM difference values in virtual models were within 1.5 mm of initial HDMM values. The variability from navigation fixed-AR is enough to consider repositioning frameless fixation without CBCT scanning for treating patients fractionated with large multiple metastases lesions (> 3 cm) who have difficulty enduring long beam-on time. This system could be applied to novel GKRS navigation for frameless fixation with reduced preparation time. It has found good potential applications in many fields, such as military training, entertainment, manufacturing, and medical in recent years. In the neurosurgery field, AR is used as a volumetric image guide3 and phone-based neurosurgical navigation system6. AR navigation uses various devices such as smartphones, desktop PCs, head-mounted displays, and AR glasses. The AR system commonly consists of an outside-in method using sensors either attached to the computer, head-mounted display, or pre-installed, which can be operated intuitively in conjunction with hand movements7, but sensor recognition could go out-of-range due to misalignment after initial registration. The inside-out tracking method can detect a continuous tracing of the target from the user’s location through cameras and sensors mounted simultaneously on the device used for visualization8. Although inside-out tracking has less accuracy compared to outside-in tracking, installing the equipment for visualization and object detection for registration is low cost. We previously reported on an inside-out tracking-based AR-neuro-navigation system using ARKit® (Apple Inc.) based software9. We applied the inside-out tracking for radiosurgery by mounting it to a 4th generation iPad Pro  to test the feasibility to develop clinically usable inside-out tracking AR in gamma knife radiosurgery (GKRS), which utilizes devices in a fixed-stated, called fixed-AR.Augmented reality (AR) is an advanced technology that mixes the virtual world with the real world in different proportions11. Accuracy with head-frame fixation is essential to limit irradiation of the surrounding anatomical structures12. Head-frame placement is invasive involving screw fixation at four specific points in the patient’s skull and difficult for fractionated treatment of large lesions13. The latest version in GammaKnife (GK) Icon™ is capable of non-invasive fixation and fractionated treatment by utilizing cone-beam computed tomography (CBCT) and detecting the motion by high-definition motion monitoring (HDMM). CBCT could be acquired by either a higher signal (CTDI 6.3) preset or lower dose (CTDI 2.5) and registered with the stereotactically-defined image set for comparison between patient coordinates at the time of treatment imaging; the HDMM system can currently be used for head immobilization with a thermoplastic mask instead of a head-frame14. It is the frameless fixation of the GK Icon™. During subsequent delivery of the adapted treatment plan from the HDMM system, it tracks the displacement of the patient’s nose marker related to the four immobile reflectors fixed to the Icon™ head support system in real time15. However, the irradiation is executed only when the magnitude of displacement returns under the threshold; if the threshold is exceeded, the new CBCT scan is processed to allow coordinates acquire a new position. Wright et al. suggested that the target and nose marker typically varies throughout a clinically relevant extent of stereotactic space, and the average HDMM threshold of 1.4 mm may be appreciated for 41 volumes15. However, patients with multiple metastases or older age seem to be intolerant to the lower threshold according to the increased beam-on time. Kim et al. reported that the elapsed beam-on time, including beam-paused time due to motion of the patient, defines the tolerance for around 30 min (min) in older patients (> 65 years)16. Thus, keeping the appropriate HDMM threshold without the new position for CBCT scanning is important for intolerant patients in GKRS frameless fixation. The navigation of patient positioning under frameless fixation is useful to reduce the preparation time and unnecessary CBCT scanning. If we used CBCT images to make a 3-dimensional (3D)-virtual model using fixed-AR, the frameless fixation could possibly be repositioned, guided by a 3D-virtual model, based on initial planning for CBCT without unnecessary CBCT scanning.The high dose of radiation delivered through GKRS requires a high degree of accuracy and immobilization of the head17. The front-facing camera provides the depth data in real-time along with visual information, and the system uses a light-emitting diode to project an irregular grid of over 30,000 infrared dots to record the depth within milliseconds. To scan objects, an additional application was installed18. Although Heges application was evaluated with the finest 3D resolutions under 0.5 mm17, the accuracy is affected by the scanning strategy and post-processing19. The potential of TrueDepth in the recent iPad Pro as 3D-scanning in Heges application was evaluated using fixed-AR in GKRS.3D-scanning increases the accuracy and makes it easy to obtain the virtual 3D-model. Recently, TrueDepth technology in the latest devices by Apple Inc. is used for measuring the task with accuracies in the millimeter range. TrueDepth uses vertical-cavity surface-emitting laser (VCSEL) technology and consists of an infrared camera, a conventional camera, proximity sensor, spot projector, and flood illuminatorTo apply this new AR technology for GKRS, the virtual models were established using existing planning CBCT images, and a novel TrueDepth 3D-scanning method. In this study, we investigated the navigation of frameless fixation using fixed-AR with the virtual models of CBCT scans and 3D-scans into a 3D-printed patient model for GKRS.When the navigation of frameless fixation was executed through the new application, it overlapped on the 3D-printed model. To complement the fixed device state, the quick response (QR) marker attached to the mask indicator and iPad Pro was installed to the cradle beside a couch bed. The QR marker was designed for various directions by adjusting the registration target. Fixed-AR navigation is confirmed to correctly overlap with the 3D-printed patient model and virtual models based on 3D-scanning and CBCT in Fig. Rotational and translational set-up errors for frameless fixation based on virtual models using fixed-AR are summarized in Table The fiducial markers in the 3D-printed model are defined in seven points out of eight points because of the exceeding image definition from CBCT scanning. The mean error of fiducial marker coordinates for frameless fixation based on virtual models in fixed-AR are summarized in Table For patients undergoing GKRS treatment with frameless fixation for a few days, we measured the HDMM differences between initial fixation and re-fixation, navigated by the virtual models for three days. The mean differences in the virtual model of 3D-scanning and CBCT were 1.19 ± 0.32 mm and 1.21 ± 0.02 mm, respectively. The re-fixation navigated by the virtual models without CBCT revealed HDMM values under 1.5 mm.The AR-guided initial preparation time was approximately 8 min whereas the CBCT rescan was approximately 5 min. If we initially set the fixed-AR, it would take only 1 min to repeat the GKRS procedure.In cases of inaccurate registration after fixed-AR navigation, the user could use the correction function to adjust the surface position of the fixed-AR navigation that accurately overlaps with the nose marker in GKRS. Both 3D rendering in 3D scanning and CBCT distorted from registration were adjusted to the correct position using the correction function.20. Frameless fixation is widely used for large lesions with hypofractionated treatment; it is important to endure the tolerance and long beam-on-time, allowing immobilization of the patient’s head. If AR navigation using virtual models replace the patient’s head position navigation, maintaining the appropriate HDMM threshold without repositioning CBCT scanning will be accomplished more easily with the reducing of the preparation time and the irradiation due to CBCT scanning in GKRS. This study represented the mean set-up errors within 1.5°, and 1 mm in both methods. The mean differences of fiducial markers from coordinates were within 2.5 mm, and the 3D errors were within 3 mm in both methods. Our fiducial marker error results were larger than set-up errors; however, the mean errors determined by manual measurement are generally acceptable within the 3 mm threshold. The mean differences of HDMM values were within 1.5 mm compared to initial HDMM values.GKRS requires an accurate and precise high-dose radiation for a specific target while minimizing the potential radiation toxicity for the surrounding tissue. Recently, the GK Icon™ version is capable of frameless (mask) fixation using CBCT, which is used to verify the patient position during set-up prior to irradiationWe demonstrated that virtual models with fixed-AR could be applied for intolerant patients with frameless fixation, requiring repeated CBCT scanning to exceed the HDMM threshold.22. In radiosurgery, all patients have various lesions including malignant or benign tumors, vascular disease, and functional diseases. In our country, demographic projections for older adults have increased, implying a consequent increase in cancer incidence and mortality in this population23. Radiosurgery delivered by highly focusing radiation with sharp dose fall-off is theoretically expected to reduce delayed neurotoxicity24. Recently, hypofractionated radiosurgery was found to be effective as a single-session radiosurgery with minimal toxicity for large brain metastases (> 10 cm3)26. However, the patients with multiple metastases or older age seem to be intolerant to the lower HDMM values according to increased beam-on time. Thus, keeping the appropriate HDMM threshold without a new CBCT scanning position is important for intolerant patients in frameless fixation in GKRS. In case of radiation exposure, although CBCT (2.5 mGy or 6.4 mGy) is a low dose CT, CBCT scans still involve radiation exposure. Furthermore, repeated CBCT scanning could require continual CBCT data storage. CBCT data is approximately 200 MB, the virtual model 3D data is approximately 80 MB. The virtual model 3D data requires less storage capacity and replace CBCT rescan data.Patient-specific planning based on individual characteristics and conditions is important for precise treatment in neurosurgery27. TrueDepth-based 3D scanning reveals the highest deviations in cylindricity (0.82 mm on average) and roundness (1.17 mm on average)17. In another 3D scanning study, Camison et al. demonstrated that the points calculated from a total of 136 distances had an average deviation (mm) of 0.8428. Our results revealed that translational errors were less than 1.00 mm in all axes in virtual models. The rotational error in the Y-axis was only higher in the virtual model of 3D-scanning compared to that of CBCT. However, all fiducial marker errors were under 3 mm without CBCT scanning. These results could be applied to the patients with metastases, long beam-on-time, and no eloquent areas. Paul et al. demonstrated that when using 3 mm as a cut-off there was no effect on local recurrences identified 29.The scanning accuracy decides the potential use of 3D scanner applications. The iOS of Apple’s smartphones and tablets provides 3D data without the operator’s measurement experiences, called TrueDepth Scanner, based on the structure-light principleThe novel method of inside-out fixed AR is performed by a physician or physicist experienced in GKRS to match the nose marker target under frameless fixation based on user-determined registration with the 3D-printed patient model and virtual models. The fixed-AR navigation do not execute the automatic registration into frameless fixation; however, the position of fixed-AR images could be adjusted using the correction function into a fixated-state in a frameless adaptor. We recommend that the navigation of the fixed-AR system could be utilized simultaneously along with the frameless-fixation before pretreatment CBCT scanning. If a patient cannot endure the long beam-on time, frameless re-fixation without CBCT can be performed with a short resting period using the fixed-AR system. Frameless fixation can be maintained for a long time, which may result in pressure on the face and make the patient feel uncomfortable. For this reason, it is important to conduct the frameless fixation close to patient's initial position. The AR navigation has the potential capability for real-time monitoring of a patient’s movement in GKRS. The detection of the patient’s movement only depends on the motion marker in GK Icon™; however, if the real-time monitoring is possible in AR navigation, frameless GKRS can be performed without a mask or with the mask loosely fixed in the future.This study has a few limitations. The fixed-AR system is not automatically registered to the object; it takes a few attempts to overlap the object and virtual models. The development of a fixed-AR system should be considered to register AR automatically into the object using real-time tracking, and we are planning to build a storage server with patient specific information, including virtual models in the fixed-AR application in order to conduct a clinical trial in the future. Although the virtual model of 3D-scanning consists of a fixed-AR system, it is still inaccurate to register it to the entire real object. 3D-scanning accuracy will determine the potential applications of 3D-scanners. To guide  the surface fixed-state with 3D-scanning, the general 3D-scanning accuracy of the entire object should be evaluated.We demonstrated that fixed-AR navigation is a useful tool for frameless fixation without CBCT for GKRS. This method using conventional equipment and fixed-AR with inside-out tracking could be directly adapted to GKRS. Overall, when planning for small lesions or eloquent areas, frameless fixation and repositioning with CBCT scanning should be considered. However, in cases of patients with large metastases, no eloquent area and continual movements under frameless fixation could be navigated well with re-fixation and repositioning using a fixed-AR system without CBCT scanning.The 3D-printed patient model was produced in the following three stages: 1) creation of a stereolithography (STL) file for 3D printing; 2) printing physibles using a 3D printer; and 3) post-processing performed via manual editing. The process was approved by the institutional review board  and used to validate set-up errors, fiducial marker coordinates, and HDMM values. The IRBs of two institutions where a case originated waived the requirement for informed consent for using simulated patient’s phantom since there was no interaction with the patient. The 3D-printed patient model had a frameless adapter with a head cushion and was fixed with a thermoplastic Nanor® mask . In the fixed-AR setting, we measured the initial fiducial marker coordinates after planning CBCT and HDMM values, which kept the motion value for 10 min, as presented in Fig. https://www.meshmixer.com) to trim the background scanning, and smoothing processes were performed to improve the pixilation and obtain more uniform triangles.The 3D-printed patient model with frameless fixation was taken for TrueDepth scanning using Heges application with iPad Pro. The STL scan file of the scan was imported to Meshmixer version 3.5 . The DICOM files were imported to MEDIP software for 3D-rendering, trimming the background, and smoothing processing. The MEDIP software is easy to operate due to its machine learning-based semi-automated segmentation function30. Thus, the 3D data can be prepared by medical staff without computer expertise. This procedure is presented in Fig. The 3D-printed patient model with frameless fixation was also scanned for planning CBCT scanning using Leksell GammaPlan Version 11.1 , and exported CBCT Digital Imaging and Communications in Medicine (DICOM) files. The authors developed and released the software  device recognition is visualized, followed by 2) QR marker recognition, and 3) AR implementation and registration within the running environment. The workflow of the inside-out AR navigation algorithm and running fixed-AR is presented in Fig. The inside-out AR navigation (METAMEDIP navigation) was developed using ARkit® (Apple Inc.). It has not been released yet, but the technical methods were described in the previous studyThe QR marker images are used for estimating the position recognized by the camera in the feature detection algorithm, which commonly considers the corner and intersection of lines or the part with clear color contrasts (Black and White) as a feature point. After the device position is determined, the QR marker, recognized by the scale of objects in virtual space, is determined by calculating the distance between the device and the QR marker from the size of the recognized marker in the preceding step. After all the requisite data are calculated, the 3D scanning or CBCT-based virtual models are displayed at the designated positions from the marker for confirmation by the user. In case of errors in the automatically processed pre-registration, the user can correct the registration error by adjusting the position of the 3D virtual model using the correction function and then fix the position of the models to complete the registration.11, which navigated to re-fixation using the fixed-AR system with the virtual models. Set-up error was defined as displacement of the skull in the stereotactic space. Setup errors were investigated by translational (mm) and rotation (°) methods for three days. The registration accuracy of the fiducial marker was validated by coordinating pretreatment CBCT, scanned from navigated re-fixation by the fixed-AR, to planning CBCT. The fiducial markers were attached to both hemispheres of the 3D-printed patient model in 8 points.The registration accuracy was measured using the following three methods: intuitive validation through visualization; set-up errors; and quantitative validation. The 3D-printed patient model was created and matched with 3D virtual models. Set-up errors were assessed by comparing the planning and pretreatment CBCTs31.We defined the error of coordinates as follows:Δx = x-axis coordinate error in planning CBCT + pretreatment CBCT or virtual models.Δy = y-axis coordinate error in planning CBCT + pretreatment CBCT or virtual models.Δz = z-axis coordinate error in planning CBCT + pretreatment CBCT or virtual models.Furthermore, the 3D error (Δr) was defined as a localization error by the following formula:The differences of HDMM values were investigated  in initial fixation and re-fixation navigated fixed-AR. The mean accuracy was evaluated by measuring the X, Y, and Z coordinates for three days. We also measured the preparation time from setup to GKRS procedure between the AR guided re-fixation and the CBCT rescan.9: the virtual space is shown on the device screen through the coordinates ; the local coordinate is a part of the virtual 3D model itself. The world coordinate refers to the virtual space wherein the model is placed, while the camera coordinate refers to the coordinate system wherein the world coordinate is viewed from the standard reference of the camera. The correction function is based on the world coordinate, and the movement is based on the camera coordinate, which assists the user in determining the desired direction and the rotation of movement intuitively.For cases when automatic registration using the QR marker could be misaligned, the correction function of fixed-AR was developed and could alter the location of the virtual 3D model of AR navigation based on the following principlesAlso, the opacity of the virtual models can be adjusted by using opacity-adjustment function. These functions allow AR implementation by further emphasizing the objects.


Unlike traditional visualization methods, augmented reality (AR) inserts virtual objects and information directly into digital representations of the real world, which makes these objects and data more easily understood and interactive. The integration of AR and GIS is a promising way to display spatial information in context. However, most existing AR-GIS applications only provide local spatial information in a fixed location, which is exposed to a set of problems, limited legibility, information clutter and the incomplete spatial relationships. In addition, the indoor space structure is complex and GPS is unavailable, so that indoor AR systems are further impeded by the limited capacity of these systems to detect and display location and semantic information. To address this problem, the localization technique for tracking the camera positions was fused by Bluetooth low energy (BLE) and pedestrian dead reckoning (PDR). The multi-sensor fusion-based algorithm employs a particle filter. Based on the direction and position of the phone, the spatial information is automatically registered onto a live camera view. The proposed algorithm extracts and matches a bounding box of the indoor map to a real world scene. Finally, the indoor map and semantic information were rendered into the real world, based on the real-time computed spatial relationship between the indoor map and live camera view. Experimental results demonstrate that the average positioning error of our approach is 1.47 m, and 80% of proposed method error is within approximately 1.8 m. The positioning result can effectively support that AR and indoor map fusion technique links rich indoor spatial information to real world scenes. The method is not only suitable for traditional tasks related to indoor navigation, but it is also promising method for crowdsourcing data collection and indoor map reconstruction. Consumer applications, like smart phones integrated with GIS, could provide users with useful information. This geospatial information however, might be difficult to understand. Data pertaining to complex indoor environments is especially challenging for users with limited geographic knowledge, but visualization enhances situational awareness among users for a more satisfying experience. Thus, in either 2D to 3D representation, flexibility and realism requirements for spatial information visualization are becoming ever more demanding. In contrast to traditional visualization methods, AR combines virtual objects with a real scene, as an intuitive way to convey information that is otherwise difficult to transmit . The intResearchers have made some useful explorations in AR-GIS . Ruta etAR and GIS techniques have received increasing attention. Thus there is a need to better understand the particulars of AR interfaces in indoor environments beyond location-awareness affecting GIS data visualization and interaction performance in AR-GIS applications. Camera pose tracking is the keystone for accurate understanding of spatial relationships in AR visualization . To overThe main challenge is that most existing AR applications are usually used for short periods of time and for specific purposes, they do not allow for a continuous, universal, context-aware AR experience, except that there is still a lack of effective organization in the dynamic spatial information visualization in AR-GIS system. It leads to spatial information clutter and limited visibility due to fixed location. Visualizing spatial data in an indoor AR environment requires special consideration because a disproportionate mixture of AR content or redundant information will mislead and confuse users. Therefore, organizing these dynamic spatial information and visual elements is another challenge.In this paper, the objective is optimizing and enriching the spatial information visualization in indoor environments. We bridge the indoor map and situational visualization with real world scene based on mobile augmented reality technique. Firstly, we divide an indoor map by a regular grid and build the index of map data to diminish redundant data and limit AR visualization within a certain range of the current location. At the camera tracking stage, we propose a novel multi-sensor fusion-based algorithm, which employs a particle filter to fuse BLE and PDR. Based on the direction and position of the phone, the extent of an indoor map visible in the camera view is extracted and the spatial entities calculated just for this area. Considering different poses of the mobile phone in users’ hand, we also designed changed forms of the AR view. After applying a coordinate transformation to the spatial entities, the fused AR-GIS system renders spatial information in relation to the area visible in the dynamic camera view.The contributions can be summarized as follows:•We propose a novel AR-GIS method for indoor environments that fuses an indoor map with dynamic situated visualizations.•We designed an online indoor positioning method that fuses the BLE and PDR for AR camera tracking.•We designed a flexible AR system that accessibly visualizes a variety of types of spatial information regardless of the pose of a mobile phone.The organization of this paper is as follows. The related works are briefly reviewed in Section 2. Section 3 discussed the main methods. The experimental results and analysis are described in Section 4. The conclusions are then presented in Section 5. The Abbreviation table is shown in AR has gradually emerged as a popular way to display LBSs. Hence, researchers are focused on AR techniques for GIS visualization in two major areas; camera tracking techniques for estimating camera poses and tracking target objects, and AR visualization to render virtual data onto a live camera view.Indoor localization enables AR camera tracking. The method based on visual features from images was representative way for AR. In order to solve this problem, SLAM is a widely solution for AR tracking. In addition, some experts consider using sensors embedded in mobile phones for locating, such as WiFi or accelerometer . InsteadAR visualization is not only about rendering computer graphics, but also can help make virtual 2D/3D data easier to understand as an aid to navigation in the real world. For helping designers in managing the aspect of layout and the representation of reasonable AR view, To solve the problem of limited information visibility, The existing AR-GIS applications only provide local spatial information in fixed location for users. The applications with comprehensive spatial information are rarely reported in the literature. These systems do not allow for a continuous and context-aware AR experience. Moreover, the dynamic spatial information visualization in AR-GIS system is tend to clutter and limited visibility due to fixed location.In this paper, we focus on how to localize and organize the spatial information into an AR view. The goal is to construct a mobile augmented reality system that visualizes spatial information in indoor environment. The proposed AR-GIS system is fused an indoor map and situational visualization based on a mobile phone, which can provide dynamic and comprehensive spatial information for users.The proposed workflow is illustrated in Since we set up a configuration to display the indoor spatial information, the first step in our implementation is tracking the camera pose based on the mobile phone. We propose a novel multi-sensor fusion-based algorithm, which utilizes a particle filter to fuse BLE and PDR. The indoor localization algorithm is presented in  The input of the algorithm includes BLE and inertial sensors data. Based on the BLE data, we obtain the BLE-based localization result. Meanwhile, based on the inertial sensors data, we get the displacement estimation result. Then, we use particle filter to fuse these two results. The displacement is used to update the particles for the particle filter.BLE is a wireless personal area network technology designed for applications in the healthcare, fitness, beacons, security, and home entertainment industries. Proximity sensing is one of its applications, which also provides a new method for indoor localization. BLE realizes indoor localization by measure the distance between a mobile device and several beacons. The locations of the beacons are known. Specifically, based on the known location of the beacon, the received signal strength (RSS) of the beacon can be used to estimate the distance from the mobile device to the beacon, which can be expressed as λ is the distance from the beacon to device, λ0 is reference distance , η is the path loss exponent, and X is a zero-mean Gaussian distribution variable with variance where i = 1, 2, …, N. N is usually more than 3. The measured distances are di. Then, we can get N equations: N equations, we can get the location of the mobile device.The localization method is called triangulation. If the location of the mobile device is Because of the complexity of indoor environments, the measured distance usually contains error, which makes localization results unreliable. We propose a novel algorithm that improves the localization accuracy of BLE. By the beacon, we can obtain the signal quality index (SQI), which reflects the confidence of estimated distance.p = (wi∗pi),, where p is the estimated location, wi is the weight, pi is the location of the ith beacon, k is a parameter, which is determined by experiment. wi is calculated based on the measured distance between the device and beacon:If the SQI is greater than the threshold, we use a triangulation method to calculate the position of the device; otherwise, we use fingerprinting-based method to estimate the location. A weight k-nearest neighbour (WKNN) is adopted for location calculation. The location is calculated by the following equation: wi = 1/di,1) wi.2) normalize PDR algorithm is utilized to estimate the displacement. If the previous location is n the step number, and the heading. Step number is obtained by the peak detection algorithm  are the parameters that can be trained offline.where l is the step length, lgorithm . The stelgorithm : l = a∗fThe PF algorithm is based on the sequential Monte Carlo framework, applied to nonlinear and non-Gaussian estimation problems . A typicInitialization: Sampling N particles based on the initial localization result.ith particle is updated by Prediction Sampling: Predict a new particle l and Δh are the distance and heading change obtained by PDR, and hk is the heading at time k.where Δ.Importance Sampling: Calculate weights Normalization and Resampling: The weights are normalized and resampled. In the resampling process, particles with low weight are deleted and particles with high weight are duplicated.In our proposed method, the initial localization is obtained by on BLE-based localization. The particle prediction is realized by PDR algorithm. The weights are calculated based on the distances between the new particles and BLE-based localization results. Based on this online indoor positioning method for AR camera tracking, we fused AR and indoor map for GIS data visualization.The workflow of AR-GIS visualization was shown in A map constructed of planar features will block the real scene; the indoor map is only composed of line feature that looks transparent in AR view. In order to display attribute information on the indoor map, we keep the text information as a texture and map it onto indoor map. Thus, textual information will align with indoor map whenever an application requires the user to zoom in, zoom out or rotate the map.The user may be distracted by showing entire map of a large space in an AR view. The displayed part of indoor map should be limited within a certain range of the current location. Thus, we divide indoor map by a regular grid and build an index for map data. Each regular grid cell records the minimum bounding box, transverse step, and longitudinal step. During the AR visualization, we get the current position grid as center, and acquired its surrounding eight grids for displaying. This data structure based on a spatial index of regular grid needs low storage and little communication.We designed the interface of AR-GIS system that fused the situation and indoor map visualization. As described in We transformed all these visual elements into the AR coordinate system. As shown in m, ym). We put the point M as a new original point, and reduce the size of indoor map in accordance with a certain proportion to keep it within the AR view. Considering the indoor map coordinates in Euclidean space , we define the AR coordinate system as . The coordinate transformation can be described as We acquired the rendered part of indoor map according to the current position, and calculated the bounding box of indoor map ABCO and the center point M of positioning resulte.We evaluated the effect of the number of particles in PF algorithm on the performance of the proposed positioning method. The performance was evaluated in two metrics: average positioning error and computing time. The computing time is a key factor since the proposed positioning system is implemented online on a smartphone. After determine number of particles, we the collected the statistical positioning information of each route. Take route 1 as an example, we captured 143 BLE location, and detected 203 steps. Based on these collected data (143+203), we obtained 210 localization results by using PF algorithm. The statistical positioning information of each route was given in As shown in We also compared the proposed method to PDR and BLE by using average positioning errors. As shown in The online positioning results of different positioning methods are shown in We can see that the PDR positioning error is large; the estimated trajectories deviate sharply from the ground truth. The BLE positioning results were more accurate that of PDR due to our proposed algorithm. However, in some areas, BLE has occasional large errors. For example, the bottom right of Route 1 and bottom left of Route 2, show significant deviations. Our proposed PDR and BLE fusion method is more accurate than either PDR or BLE alone. The estimated trajectories are consistent with the ground truth.https://youtu.be/XNhpaIWk1IQ.We walked with the smartphone to the test the AR-GIS visualization, and recorded the spatial information visualization results. The video of mobile augmented reality system was uploaded on the website: The AR visualization effects are illustrated in The second row , 14F shoWe discuss our approach as compared with two state-of-the-art AR systems: . Xu et aIn this paper, we designed a method for fusing AR view and indoor map in changing environments during AR-GIS visualization that also considers the changes of the AR view when the pose of the mobile phone shifts. We presented an innovative indoor positioning approach that fuses BLE and PDR to enhance the accuracy of AR camera tracking. The experiments in an office building of Shenzhen University demonstrate that dynamic AR-GIS visualization techniques can display rich spatial information in real time on mobile phones, while preserving a high accuracy in the AR-GIS fusion. The AR content includes current position, indoor map and spatial information, which change as the real scene changes. The layout of spatial information is rendered reasonable in AR view. This AR-GIS visualization will be clear and easy to understand, which effectively optimizes and enriches the spatial information in the visualization of indoor environment.Though the AR-GIS system is shown indoor map onto the AR view, it can’t adapt to the change of the scene size. Thus, in future work, we will try to process indoor map data and improve the visualization technique method to realize the adaptive AR visualization. Besides, the current application only labels the name of indoor scene, which can be further advanced with more details about the spatial information. Furthermore, the smoothing algorithm can be applied to enhance the accuracy of the device’s position.10.7717/peerj-cs.704/supp-1Supplemental Information 1Click here for additional data file.10.7717/peerj-cs.704/supp-2Supplemental Information 2This is an online indoor positioning method that fuses the Bluetooth low energy (BLE) and pedestrian dead-reckoning (PDR) for AR camera tracking. The read_json.m is reading the pdr and BLE data for positoning and build Route 1. The read_json2.m is reading the pdr and BLE data for positoning and build Route 2. The read_json3.m is reading the pdr and BLE data for positoning and build Route 3. The read_json4.m is reading the pdr and BLE data for positoning and build Route 4. The plotline1.m is use for drawing the CDF figure.Click here for additional data file.10.7717/peerj-cs.704/supp-3Supplemental Information 3Click here for additional data file.