Three decades after the first set of work on Medical Augmented Reality (MAR) was presented to the international community, and ten years after the deployment of the first MAR solutions into operating rooms, its exact definition, basic components, systematic design, and validation still lack a detailed discussion. This paper defines the basic components of any Augmented Reality (AR) solution and extends them to exemplary Medical Augmented Reality Systems (MARS). We use some of the original MARS applications developed at the Chair for Computer Aided Medical Procedures and deployed into medical schools for teaching anatomy and into operating rooms for telemedicine and surgical guidance throughout the last decades to identify the corresponding basic components. In this regard, the paper is not discussing all past or existing solutions but only aims at defining the principle components and discussing the particular domain modeling for MAR and its design-development-validation process, and providing exemplary cases through the past in-house developments of such solutions. The reflected sky made the augmentation more natural and provided a convincing perceptual rendering of the augmented scene. It is fascinating to see that the components of this Renaissance-era system designed, built, and validated by Brunelleschi closely match the modern ones presented here.Probably the first construction of an Augmented Reality (AR) system goes back to Filippo Brunelleschi around 1413 ,3, BruneWhile many definitions of AR have been given throughout the last decades, we will try to provide one inspired by all these definitions and try not only to consider the impact of novel technologies and concepts but also bring in basic philosophical discussions about the current and future role of AR in education, training, and performance of a variety of personal and professional tasks. We furthermore point the reader to the literature review about Medical Augmented Reality (MAR) by Sielhorst et al. from 2008  and a mo1Physical-World Modeling2Display, including Augmentation and Perceptual Rendering3Interaction4Evaluation, including functional Testing & Validation as well as EthicsOne of the most commonly applied definitions is the one proposed by Ron Azuma in 1997 . Most deLet us focus on the role MAR could or should play in our society, particularly in healthcare-related fields. Due to the recent commercial availability of AR display systems, many clinical, academic, and industrial labs have focused on low-hanging fruits aiming at the superimposition of virtual to real objects, often for navigation purposes. We believe, however, that the community will move to more intelligent AR solutions, enabling higher levels of understanding and mental mapping rather than the direct superimposition of an insertion path or a deep-seated target onto the expert’s view of the surgical scene. Since the time of Aristotle, we know that humans use all of their senses to explore the environment. Humans interact with their environment and remember experiences, which gradually turn into intuitions: When you look at a wine glass, you know that it can break. None of your senses are able to detect this, but you infer this familiarity from past observations and experiences. These experiences are, in turn, based on your human senses. AR could allow us to make experiences based on much more powerful sensing than our limited human sensing capabilities, accumulate greater knowledge, and let it turn into intuitions. The role of AR, therefore, becomes at least twofold: On one hand, it fetches and brings relevant information to our direct attention. On the other hand, AR allows us to build a deeper understanding of our environment to perform better, even without direct augmentation, as we have learned and created a better understanding. This shows what an important tool MAR can be in teaching and training not only for clinicians and clinical staff but also for the population who strives for better and more in-depth knowledge of their own anatomy and physiology and the changes it goes through during our lifetime.In addition to the multi-modal sensing and digital reconstruction of the physical world and its perceptual augmentation, a major component of any Medical Augmented Reality System (MARS) is its user-centered, dynamic, and continuous evaluation. Since AR augments human actors, its design, development, and deployment must include constant and dynamic user evaluation. The best MARS can be developed not only after in-depth use case studies but also through the dynamic participation of users in each step of its iterative design and development. It is important to notice that scientists or clinicians can only evaluate one particularly designed solution. It is generally impossible to evaluate AR as a whole. Therefore, one has to be very careful about publications claiming to “evaluate AR” or its effect on users while they are only evaluating their particular system and methodology.Finally, ethics needs to be considered from the very first moments of problem definition throughout the iterative design, development, evaluation, and deployment process. This is crucial in particular for MARS, in which not only the well-being of an individual patient is at stake, but potentially the healthcare of human society and its future development.In the early 1990s, Bajura et al. [In the following sections, we introduce the proposed Medical Augmented Reality Framework, give a detailed description of all of its components, and showcase exemplary MARS developed by the Chair for Computer Aided Medical Procedures and its clinical, academic and industrial partners, and discuss their basic components in light of the framework.Digital World, (ii) the AR/VR Display, and the (iii) AR/VR User Interaction component. The system interacts with the Physical World by creating a digital representation of its environment and interacting with the user through a dynamic UI. As an overarching methodology, a dynamic and continuous Evaluation forms an essential part of the conceptualization, development, and translation of an MARS, covering aspects from system validation to ethics and regulatory matters. In the following sections, we will explain all components of this framework in detail.To define MAR and enable the systematic design and validation of these systems, we propose a framework as illustrated in AR is characterized by enabling the co-localization and display of digital and physical information in the physical space, realized through key technologies such as spatial computing. The physical world is not a component of an MARS. Still, the system needs to sense and understand its environment to have both content related to the physical world and a canvas to represent it.The users are part of the physical world, and the focus of the combination of real and virtual content, so that they perceive the augmentations through their own senses.In a medical environment, the physical world comes with particular challenges.The operating room (OR) is a highly dynamic and crowded environment with high-intensity lighting conditions which, for example, influence the inside-out tracking systems of AR-enabled devices. Furthermore, additional sensing devices can capture aspects of the physical world that humans cannot perceive, e.g., the internals of a human body, using medical imaging or additional sensor data from other external medical equipment.For an AR system to create a meaningful augmentation of the physical world, it must be able to reconstruct and interpret it digitally. Several companies in the AR field express the goal of creating digital twins of the environment, e.g., entire cities or even another layer of the universe utilizing the enriching interconnectedness of the internet . An AR sCommercial AR-enabled devices today were often designed for the mass market and, therefore, usually include applications that are companions to the user in several activities: Users can access their emails, be notified about appointments in their calendars, have a voice-controlled assistant obtain information from the internet, or stream videos into the user’s AR environment. An MARS, in particular, if used for dynamic clinical decision-making, not only does not require some of these applications but may even need to prevent their installation due to data protection requirements  and patient safety . Clinical devices are often optimized for their consistent and continuous use within medical procedures and are solely used to improve clinical decision-making and healthcare delivery. MAR solutions, which are not made for teaching, training, or patient rehabilitation, but are used within routine clinical procedures, need to focus only on improving them.Like its users, the technical system perceives its environment through several sensors to create an internal representation, which is separate from the physical environment but shaped by the gathered information. Such sensors, e.g., optical, acoustic, haptic, or inertial sensors, can acquire similar information as human senses do. However, they can also observe the environment in more advanced forms than human sensing, translating originally unavailable information for humans to sense and observe beyond their natural abilities. CT, MRI, ultrasound, X-ray, OCT, fluorescence, and hyper-spectral imaging are a few of such sensors used in the medical domain.The data of the sensors must be calibrated and synchronized. The fused data from the sensors must then be organized before they can be interpreted. Thanks to advancements in computational power and intelligent analysis of the available data, the digital representations of the patient and environment can revolutionize all divisions of the medical domain, from teaching and training to diagnosis, treatment, and follow-ups, to patient information, preventive actions, and rehabilitation.The sensor data need to be intelligently processed to enable the digital system to interpret the collected input. In the case of marker-based AR, the system mainly perceives the identity and pose of markers positioned within the environment allowing relative positioning of virtual objects. Feature-based systems enable tracking and localization using natural features. However, the virtual objects were often placed at predefined locations, defined by the AR system designers, relative to the natural feature. Machine Learning and Artificial Intelligence start enabling semantic understanding of scenes, their components, and their interaction within the environment. This could allow AR systems to have higher levels of perceptual understanding of the environment they act in. In some of the most recent work in surgical scene understanding, Ozsoy et al. ,12 used Digital World. Here objects of the physical world relate to their digital counterparts. An AR architect can author specific virtual objects with complex behaviors within this digital world. We can, for example, specify a particular virtual object to be attached to a physical one or define behaviors of virtual objects based on changes in the physical world.The above processes lead to a digital representation of the environment, which we call the Advances in computer vision and Machine Learning, as discussed in the previous section, give us the hope that in the near future, semantic understanding of the surgical environment, its main components, their roles, and interactions may allow for a higher level of AR design and system commands, e.g., visualize and augment the endoscopic view of surgery on the trainee’s headset only during coagulation performed by the head surgeon.https://www.medicalaugmentedreality.org/mar2021.html, accessed on 1 November 2022).The mere modeling and reconstruction of a digital world based on the physical world does not yet lead to an AR system. An AR system must convert this internal representation into messages of valuable augmentations that it can convey to its user. This is achieved by rendering the augmented digital world in a way its users understand. Rendering and displaying in AR are not restricted to creating images displayed on a screen. Besides visual rendering, one can also include audio or haptic rendering. Some of the creative participants of the Medical Augmented Reality Summer School 2021 even proposed to use gustatory sensing to pass pertinent messages to surgeons during minimally invasive surgery  to interact with machines had already been designed before the concept of user interaction was formalized. Historically, with every advance in computing, new display forms, new input methods, and new interactions were introduced. We saw decidedly different classes of interaction methods coming to fruition roughly every 20 years: Command-line UIs were the dominant method of user interaction in the 1960s and 1970s, where users had to type explicit commands on a keyboard. In the 1980s, computers became more user-friendly with graphical UIs and mouse input devices introduced to the general public. UIs continued to evolve in the 2000s when mobile screens popularized multi-touch controls. In the 2020s, the emergence of commercially available AR systems led to limited commercial deployment of AR solutions for various applications, including industrial maintenance, education, and healthcare.Established input devices do not seem to be suitable for most AR applications. The community needs to take advantage of rich sensing and advanced algorithms and develop novel user interaction paradigms to make interacting with such systems more natural and context-driven. E.g., multi-modal hand gesture input, eye or head gaze, voice commands, body motion, and AR controllers could serve as more instinctual interactions. While promising implementations have been presented, effective natural interaction solutions are still to be presented and turned into novel standards.User interaction in the medical domain comes with even more challenges, such as sterility requirements in the OR, technical regulations for incorporating exercises for patient rehabilitation, or the need for high accuracy, e.g., when augmenting precision surgeries. Novel and efficient user interaction paradigms for MAR solutions, particularly in a surgical environment, are still to be developed.adaptive, believable, and consistent: Adaptive: AR interactions become instinctual when they are coupled with the context of the user’s environment, task, action, and mental state. Context is not always known at design time. Therefore, UIs need to be adaptive [Believable: Physically-based rendering greatly impacts perceived realism. However, the augmented content does not always need to be realistic. In some applications, it may be necessary for the user to distinguish between the real world and its augmentation. In such cases, too high realism could confuse what is real and what is not. Consistent: With imprecise inputs from human motion , controllers, and the environment condition, captured by the multi-modal sensors, a computationally driven model is required to process the rich sensing data and turn them into consistent feedback within the AR interface. Once again, novel interaction paradigms may be required to take full advantage of mixed reality. The interactive virtual mirror paradigm we designed and developed is one of the unique interaction solutions explicitly developed for MARS [Compelling AR interfaces need to be: adaptive . In AR-afor MARS ,26,27.The users are the focus of all AR systems and should be considered accordingly during conception and development as it is their perceptual system that receives virtual content from the AR system. With a successful AR system, users can use this information to improve their interaction with their natural environment. One example of the iterative inclusion of the user towards the development of a successful system is a project conducted within our lab that integrated VR and AR for intraoperative single-photon emission computerized tomography (SPECT), which was started in 2007  and resuHence, acquiring and evaluating usability-related measurements within an AR system is essential in the development process. This observation holds even more true for critical applications such as medical treatment and surgical interventions. Regarding the visual fidelity of AR systems, consideration of human perception and intuitive behavior are essential to enabling immersive and user-friendly experiences. Since intuition may differ widely between humans (even more so when they come from different areas of expertise), designers of AR systems may often take different approaches to solve problems than the ones preferred by the final users, particularly in highly-specialized fields. Introducing methods, paradigms, interactions, novel renderings, or displays that contradict the final users’ intuition may entail an adaptation process to redefine what the users perceive as familiar. Therefore, intermediate and iterative validations involving the final users should be conducted in parallel to the conception and development phases of MARS and not only in the early or final stages.The integration of user studies during the design and development stages allows for iterative refinement of the systems. Further, it provides insight into the feasibility of integrating AR into medical procedures. These studies should be carefully designed, conducted, and analyzed. Otherwise, they can produce misleading results, wrongly reject valuable concepts, or lead to wrong conclusions regarding how these systems will be adopted by the end-users . For thiThe acquisition of qualitative measures allows for investigating the systems’ usability, including attributes such as learnability, efficiency, intuitiveness, and/or satisfaction. In addition to collecting experiential feedback from individual medical partners, using surveys or questionnaires is a common approach for gathering qualitative measures and user feedback. The content of the questionnaire depends on the type of study evaluated. Examples of such may include general surveys to gather usability (System Usability Score ), mentalIn addition to the qualitative data, the collection of quantitative data can contribute to investigating the feasibility of integrating the systems during the performance of medical tasks. Such data can determine the system’s accuracy, precision, repeatability, latency, reliability, and stability; and also play a determinant role during the certification process of the system. Quantitative results can be used to investigate if an MARS contributes to reducing the frequency or magnitude of errors, enhancing dexterity, or minimizing the time required to complete a task. These measurements also contribute to understanding how the user benefits from interacting with the system, comparing the efficacy of novel interaction paradigms, or studying how these systems can support collaboration, communication, or task performance in the OR. Therefore, the evaluation frameworks should be designed to produce reliable and repeatable data that reflect the users’ performance while interacting with the system.Once again, ethics needs to be considered from the very first moment of problem definition and throughout the iterative design-development-validation process. The final solutions of MAR need to be accessible to and affordable for all subjects whose data contribute to the design and validation of such solutions. MAR needs to target the well-being of all of society, particularly when public funds sponsor such research. It is highly recommendable to have experts of ethics personally participate in all MAR projects as team members and for all researchers involved in MAR to take explicit training on ethics of design and usability studies.The following sections show selected systems developed over two decades at the Chair of Computer Aided Medical Procedures, for which we dissect their components and map them into the framework. We argue that the proposed MAR framework can describe all MARS from the past, present, and future.3D Telepresence allows users to collaborate over distances, for example, by sharing a physical space as a real-time reconstruction with remote users connected via secure networks. Remote users may virtually join local users through a VR headset, can communicate through verbal and non-verbal cues, and interact with each other through AR and VR.As a medical use case, we identified the need for 3D teleconsultation during the first response for accidents and emergencies. E.g., in the case of massive casualties  or rare Augmented Reality Teleconsultation for Medicine (ArTekMed) that fully utilizes the proposed MAR framework, as seen in We present Visualization and interaction paradigms that are unique to shared virtual spaces seen in , such asPhysical World of ArTekMed: The physical world includes the patient with the surrounding environment and objects, comprising local users such as paramedics and bystanders.Computer Sensors of ArTekMed: On the patient’s side, the RGB-D cameras, the built-in sensors of the AR-HMD, and microphones. On the remote user’s side, infrared-based sensors capture the LED constellation on the VR headset, controllers, and body tracking pugs.Computer Perception of ArTekMed: The computation units connected to the sensors interpret the acquired data. In particular, the inside-out sensors on the AR headset use simultaneous localization and mapping (SLAM) to compute the local user’s first-person perspective correctly. Furthermore, on the VR side, the tracking system returns the six degrees of freedom pose within the digital world.Digital World of ArTekMed: The digital world consists of the reconstructed point cloud and virtual avatars to represent the users for conveying non-verbal communication and social presence. Additional tools to aid the consultation, including 3D annotations and the Magnorama, are part of this digital world and are rendered to all participating users.Rendering Displays of ArTekMed: The local users wear AR-HMDs for in-situ augmentations inside the real environment. They allow the local user to perceive the real world alongside digital visual and auditive augmentations that ArTekMed generates from co-located and remote-connected participants. In addition, the VR users perceive the reconstructed point cloud within their VR headsets and can talk and listen to the local AR users.Augmentations of ArTekMed: Augmented components are virtual elements such as 3D annotations, the Magnorama, and the avatars. To fully utilize Augmentations, ArTekMed uses the SLAM reconstruction of the AR-HMD for occlusion handling in the real environment.Human Perception within ArTekMed: Occlusion handling with the SLAM reconstruction in AR allows users to understand the depth of virtual objects concerning real objects quickly. Stereoscopic rendering within the HMDs in AR and VR lets users observe the scene in 3D. Moreover, tracking of the HMD qualifies motion parallax as an additional visual cue for depth perception.Dynamic UI in ArTekMed: Conventional 2D in 3D UIs, such as menus and a radial menu to adjust settings and tools, allow users to interact with the system on a high level. Users interact with the ArTekMed system using their bodies and transform the point cloud reconstruction or digital avatar respectively for non-verbal communication. While interacting with the environment, diegetic virtual elements such as the 3D annotations and Magnorama are part of the digital world and fused with the real world in AR.Evaluation in ArTekMed: Novel systems such as the ArTekMed disrupt standard practices in Healthcare. Evaluation should therefore cover the fundamental acceptance of every component in the system and their usability. The evaluation covers the acceptance of teleconsultation versus conventional video calls [eo calls , user reeo calls , and adveo calls ,43 - allFrom our description, we map the components of ArTekMed to our MAR framework:ArTekMed combines digital sensors, AR/VR displays, and novel 3D user interfaces virtuously and intuitively. Furthermore, the augmentations in the form of 3D annotations in space and interactive aids for navigating and manipulating 3D space demonstrate the many-faceted synergy between the digital world, the user, and the interface.In ophthalmic surgery, AR has the potential to optimize perception and understanding of the surgical scene. Conventionally, a surgical microscope provides visual access to the surgical scene, allowing visualization of the backside of the eye and an overhead view of the retina. Due to the fixed top view of the surgical site, depth is mainly inferred by shadows of surgical instruments projected onto the retina by a handheld light pipe. Today, multi-modal imaging has become available, providing complex multi-resolution imaging. In state-of-the-art setups, surgical sensing consists of 2D and 3D intraoperative Optical Coherence Tomography (iOCT) cross-sections along with the conventional microscopic view. This additional imaging provides high-resolution depth information of surgical instruments and anatomical structures and can be leveraged to extract semantic scene understanding and other helpful information for the surgeon. However, simultaneously providing all the available data to the surgeon may create high mental demand and visual overload, as surgeons must focus on various areas of multiple modalities, and mapping between these modalities is challenging.Hence, there is a need for AR systems that can merge the data in an intelligent way by integrating the key elements of each modality and blending them such that a single image is generated that shows all the necessary information to the surgeon. This, in turn, will reduce the cognitive load on the surgeon and avoid distractions that occur when switching the gaze to look at other modalities. An example of a system fusing information from multiple sources in a fused overlay is the relevance-based visualization of X-Ray images in the camera space proposed by Pauly et al. .Currently, there are two proposed approaches to support ophthalmic procedures: (i) visual AR, which directly augments the displayed imaging data to improve scene understanding and visual perception, and (ii) auditive AR, which leverages extracted information from data streams as parameters to generate or modify sound signals. The first work towards visual AR in ophthalmology was proposed in 2015 , in whicMoreover, auditive AR can also be an attractive way to convey supportive information in ophthalmic surgery while not overloading the surgical view with visual information. The works ,22 relatPhysical World in Ophthalmology: The operating environment consists of a surgical microscope providing direct visual access to the ocular anatomy of the patient. The surgeon uses both hands to manipulate micro-surgical instruments and feet to control the microscope as well as the iOCT system via pedals on the floor. In modern setups, visualization of the operating area is provided by 3D monitors next to the patient site.Sensors in Ophthalmology:  operating microscopes providing a stereo view, intraoperative OCT lasers for 2D and 3D depth visualization.Computer Perception in Ophthalmic Applications: The operating microscope provides stereo RGB images, while 2D cross-sectional slices are acquired by the iOCT system. The compounding of these slices enables 3D, and in state-of-the-art systems, even temporal 4D visualization of the surgical area.Digital World in Ophthalmic Applications: The digital world consists of the raw imaging data of all sensors brought into a common coordinate frame, as well as the semantic information of anatomical structures and surgical instruments and information about their relationship. It can further contain information about the surgical phases.Augmentations in Ophthalmic Applications: Augmentations are either integrated into the surgical microscope or the 3D display or provided via audio signals and sonification methods. In both cases, they leverage semantic understanding provided by the digital world and aim to improve the perception of the surgical scene.Human Perception in Ophthalmic Applications: Depth and distance perception in iOCT volume renderings is mainly provided by color transfer functions and by generating sound signals or modifying musical pieces.UI in Ophthalmic Applications: The surgeon’s hands manipulate the light guide and tools. Access to the surgical microscope and iOCT system, hence, also the interaction with an AR system is mainly provided via foot switches. Automatic surgical phase understanding could improve the user interface for AR systems and reduce the cognitive load on the surgeon. The design of a user-centric UI is of utmost importance, allowing optimal usability of the system and optimal perception of the provided information without disturbing the surgeon’s workflow.Evaluation in Ophthalmic Applications: Ophthalmic applications are carefully evaluated on phantom eye models, in ex-vivo animal wet lab settings, and on surgical simulators. Close collaborations with surgeons during all stages of design, development, testing, and validation are required for ethical development. Only at a later stage in vivo animal studies will be conducted, and only then can systems be integrated into clinical studies on humans.The motivation for proposing the Camera-Augmented Mobile C-arm (CAMC) ,50 is toPhysical World in CAMC: The main component of the physical world for CAMC is the deep-seated patient’s anatomy, which is sensed using a mobile X-ray system: Then, the visible patient surface, the surgical tools, the surgeon’s hand, and in later stages, the operating table, assistants, and the rest of the operating room. In the first versions of the CAMC, the co-registration of X-ray sensing and optical imaging improved surgical viewing. Later, the system got extended to include augmentation of full operating room interactions.Computer Sensors and Perception in CAMC: The sensors attached to the mobile C-arm have evolved over time. The first iteration started with a CCD camera near the X-ray source [y source . With thy source  fully rey source ,53. A rey source ,55 both Digital World in CAMC: With each iteration of the system, CAMC creates different understandings of the digital world. However, the core aspect of the system is to associate a spatial-temporal relation of the imaging data provided by the C-arm, the patient anatomy captured by the attached cameras, the surgeon, and tools in the same coordinate frame [te frame .Augmentations in CAMC Applications: Patient’s pre-operative or intra-operative 2D and 3D medical data are augmented and fused with the live optical information of the patient. The system can further visualize the trajectory of tools and annotated points, lines, and planes.Human Perception in CAMC Applications: Machine Learning improves the perception of the scene by identifying relevant objects in both X-ray and optical images captured with the CAMC system to build a fused image for better handling occlusion [cclusion . In the cclusion . In addicclusion . With ARUI in CAMC Applications: With the HMD variant of the CAMC system, the user interaction is mostly through hand gestures and voice commands [commands . The useEvaluation in CAMC Applications: The system was first evaluated on phantoms [phantoms ,61, thenphantoms ,63 and fphantoms ,65,66. Qphantoms ,63,67,68Mirrors provide one of the most intuitive concepts for humans to understand the view from a different angle. Utilizing the mirror paradigm, we incorporate AR to facilitate the exploration of anatomy in the user’s augmented mirror reflection.Magic Mirror [The c Mirror ,71,72,73Physical World of Magic Mirrors: The physical world relevant to the Magic Mirror consists of its users. To complete the mirror view, the Magic Mirror additionally captures the environment behind the users.Computer Sensors and Perception for Magic Mirrors: An RGB-D camera facilitates both the mirror view and computes the pose of the users.Digital World of Magic Mirrors: The internal representation of the system consists of the user’s body pose and an anatomical model. The user’s pose deforms the virtual representation of their anatomy accordingly.Rendering Displays of Magic Mirrors: The Magic Mirror renders the image of its color camera on a large 2D monitor.Augmentation of Magic Mirrors: The virtual anatomies are augmented onto the color image. The system further enhances the illusion of looking into the body instead of seeing a simple overlay by utilizing a soft fall-off at the transition between the color image and the augmented view based on findings of Bichlmeier et al. [r et al. .Dynamic UI of Magic Mirrors: As the anatomy augments the viewing users, they may use proprioception to move their hand to a specific organ with the visuo-proprioceptive feedback loop provided by the mirror and, once arrived, feel the location on their own body.Evaluation of our Magic Mirror: We evaluated the Magic Mirror with over 1000 medical students over multiple user studies, i.e., for Radiology [adiology  and Anatadiology . Furtheradiology .The Magic Mirror follows the principles of the MAR framework as follows:The Magic Mirror highlights how AR can be utilized for education . IdeallyDecades of active research in AR and related areas were needed to enable the deployment of the first medical products using AR to support surgeons in the OR . The curBased on the experience we gathered during many years of active research and development, we propose a conceptual framework for MARS, emphasizing the necessity of adaptive interfaces and continuous evaluation with the target audience. Finally, we show the applicability of the proposed framework by modeling four MARS from our research group, which were all developed and evaluated in close collaboration with medical experts. We believe that building AR-based support systems for healthcare professionals requires an interdisciplinary approach, where state-of-the-art methods from computer vision, computer graphics, scene understanding, human-machine interaction, sensors, and displays need to be perfectly blended into a tailored experience that focuses on relevant information for the task and is intuitive to use. While we aim to model MARS specifically, many concepts and observations apply to AR systems in other domains.Driven by the latest advances in machine learning, we expect to see more sophisticated methods for environment understanding, activity recognition, and surgical process tracking. These methods will need to be optimized to work in challenging healthcare delivery environments and to perform in real time so that developers and designers of MARS can build adaptive, context-aware user interfaces. Furthermore, we expect to see specialized solutions for distinct medical procedures that optimally combine the required sensing, display, processing, and interaction tailored to the needs of each stakeholder in the team.This proposed framework summarizes the experiences we gained from years of creating MARS and teaching about the topic, and we hope that the community can learn from these insights and categorization and use it as a springboard to transform medical environments.


In this paper, we present ARMIA: a sensorized arm wearable that includes a combination of inertial and sEMG sensors to interact with serious games in telerehabilitation setups. This device reduces the cost of robotic assistance technologies to be affordable for end-users at home and at rehabilitation centers. Hardware and acquisition software specifications are described together with potential applications of ARMIA in real-life rehabilitation scenarios. A detailed comparison with similar medical technologies is provided, with a specific focus on wearable devices and virtual and augmented reality approaches. The potential advantages of the proposed device are also described showing that ARMIA could provide similar, if not better, the effectivity of physical therapy as well as giving the possibility of home-based rehabilitation. The number of people with motor disabilities has importantly increased in the last few years due to global aging and the general improvement of clinical care and health technology. For this reason, public and private health care systems are investing more in rehabilitation technologies. Recent events, such as the COVID-19 pandemic have stressed the importance of telerehabilitation both at home and at clinical facilities . The useUpper limb motor impairment is one of the most limiting conditions for activities of daily living (ADLs), so an efficient rehabilitation is critical to recovering quality of life. This impairment can be caused by a variety of neuromuscular conditions such as stroke, spinal cord injury, neurodegenerative diseases, mistakes during surgery or aging. In the last few decades, a variety of robot-assisted technologies have been developed to outperform the efficacy of conventional manual therapies ,4. End-eAnother key factor in upper limb motor rehabilitation is patient motivation. Studies on this behalf have been explored for many years to conclude that any motivational input increases the involvement of the patient in the requested motor tasks. For this purpose, a very common approach is the use of serious games, which are therapeutic leisure virtual activities to promote engagement. There are many applications of serious games to upper limb rehabilitation . These aIn addition to robot assistance, a good method to interact with these virtual environments is the sensorization of the patient. In the first stage, a common and simple way of performing this was using accelerometers to measure arm kinematics at the sagittal plane , but thiIn this paper, we present ARMIA: a sensorized arm wearable that includes a combination of inertial and sEMG sensors to interact with serious games in telerehabilitation setups. This device reduces the cost of robotic assistance technologies to be affordable for end-users at home and at rehabilitation centers. Hardware and acquisition software specifications are described together with potential applications of ARMIA in real-life rehabilitation scenarios. Similar wearable devices have already been applied in the telerehabilitation of specific body parts such as fingers  or the eARMIA is a sensorized sleeve that allows measuring kinematics and muscular activity during upper limb movements . In ordebiceps, triceps and pronator teres. These sensors are used to determine the signal amplitude (contraction force) and other factors such as muscular fatigue. The instrumentation of these sensors is critical, as the bipolar electrodes must be placed parallel to the muscle fibers close to the middle of the muscle belly, and the reference electrode must be firmly in contact with the skin.Muscular activity is measured from three of the main muscles in charge of upper limb movements: By considering these previous aspects, all hardware elements were selected according to a series of implementation requirements for the sleeve. The sleeve needs to be easily adapted to a variable morphology, and sensors, electronics and connections should be protected from external harm. All these components were selected from low-cost and lightweight options to make the device affordable and ergonomic. The different parts of ARMIA are described next.In order to acquire all analogical signals from the device, an ESP32 board (Espressif Systems) was chosen. ESP32 is an SoC (System on Chip) that can perform as a complete standalone system to connect with multiple peripheral devices, including a variety of sensors, and has very low energy consumption. This element communicates wirelessly with the computer in order to acquire the data from the inertial and EMG sensors.Muscular information  is acquired via surface bipolar electrodes using MyoWare sensors (Sparkfun). These sensors provide a compact approach to the analysis of muscular activity and include the possibility of recording both raw signals and preprocessed signals (rectification and linear envelope). Kinematic information is acquired via Inertial Measurement Units (IMUs) (model LSM6DS33). Each sensor provides accelerometer and gyroscope data and allows measuring both orientation and relative position of arm joints in space.The overall power consumption of each element was analyzed to determine the proper power supply to all the circuitry and sensors. The power supply needs to guarantee wireless communication for a period of several hours. A two-cell Li-Po battery was finally selected due to its small size and weight. This type of battery also provides a higher capacity compared to other options and allows the system to work for 3 h at full charge. Additionally, a DC–DC reductor was included to adapt the battery voltage output (7.4 V to 5 V) to feed the microcontroller and all the connected components.Three different elements were designed to support the electrical components of the sleeve . CAD modIn order to design an adequate garment to hold the sensors, morphological restrictions were applied in the selection of the textile and size of the garment. The garment consists of two main elements: the sleeve that covers de arm up to the shoulder and the chest band to fix the wearable and hold the main electronics see . On the For the initial prototype, we assumed an average sleeve length of 32 to 33 inches (around 81–84 cm) for men and between 30 and 31 inches (around 76–79 cm) for women. The first prototype was built under this assumption. However, a wider range of the garment size will be developed to cover different populations, including children.ARMIA software is structured into three different levels. The low-level layer establishes the physical communication protocols between the different sensors and the microcontroller and the wireless communication between the microcontroller and the computer. The medium-level layer manages the data acquisition and storage from the sensors. Finally, the high-level layer develops processing and visualization modules.This layer manages communication between the sensors and the ESP32 microcontroller and between ESP32 and the computer. IMUs use an I2C communication protocol. The original library was modified to allow working with multiple addresses without the need for a multiplexor. EMG sensors already offer an analog output, so they are directly connected to the microcontroller’s analog inputs. Communication with the computer is performed wirelessly using a WiFi UDP protocol due to its simplicity. This protocol allows a high-frequency information exchange necessary for real-time analysis of kinematic and muscular data.Once all sensor information is captured by the ESP32 and sent to the computer, a database manages all the data through a visual interface. This interface offers three options: “Capture”, “Query” and “Exit”. “Capture” starts data acquisition through the UDP socket and stores all the data in the database. It also implements user management options, including parameters such as user name or capture time. Additionally, the received data are sent to another port to allow an external application to, for instance, visualize them. “Query” is used to search for previously stored information. It is possible to search data by user name or date. Once selected, CSV files are generated with all the requested data. Finally, “Exit” ends the execution of the acquisition interface.Measurements from the different sensors are sent to the computer and then processed and visualized in real time. In the case of inertial sensors, the processing deals with information from both the gyroscope and the accelerometer included in the IMU. Accelerometers are not sensitive to external accelerations not originating from gravity, while gyroscopes have a significant drift in time. In order to reduce these errors, a complementary filter was designed following Equation (1):In order to validate the first prototype of ARMIA, several experiments were performed. The main goal was to evaluate how accurate are the recorded signals from the prototype sensors and how they can be translated into meaningful inputs for the virtual activities. A description of some of the current virtual activities proposed to test ARMIA is presented in Experiment 1 consisted of tracking a specific angle profile by performing arm flexion movements. The duration of the run was 60 s. Four different angle levels were evaluated: 0° (full extension), 30°, 60° and 90° (arm flexed). Transitions between levels were interpolated to maintain the continuity of the movement;biceps muscle at full capacity during a recording of 40 s. Participants were given timing feedback and asked to perform the contractions after the first 5 s. For evaluation purposes, the first and last 5 s were removed.Experiment 2 consisted of performing five short contractions of the Five subjects participated in the study . None of them had neuromuscular disorders that could affect the recordings, and all of them signed the corresponding informed consent according to the declaration of Helsinki. Data protection documents were also signed. Two experiments were performed:biceps. As observed in In this section, we addressed current issues in similar rehabilitation technologies and described the target population of our approach. We further analyzed other tools to enhance physical therapy that can be combined with our device. Particularly, we focused on the use of Virtual Reality (VR) and Augmented Reality (AR) to complement the rehabilitation activities with ARMIA. Finally, we described the in-progress and future work proposed to improve the whole rehabilitation system.Previous technologies are commonly relying on robotic support to enhance rehabilitation intensity and dose. Rehabilitation robots have optimal repeatability, multiple programs, high reliability and are ideal for large hospitals, but their cost is unbearable for small clinical centers or patient associations, not to mention their use by particulars. These devices are also not portable, and rehabilitation setups are fixed to a particular setting with specialized personnel, not allowing their use for telerehabilitation. In order to solve these limitations, we proposed the use of wearable technology such as ARMIA, which has proven to be a less cumbersome and economical way to approach physical rehabilitation ,22.Many wearable alternatives have been developed in recent years to provide ways of rehabilitating upper limb function from home. An interesting approach is GameRehab, a telerehabilitation system that combines serious games and wearable kinematic sensors . Other sThe proposed device ARMIA combines the advantages of previously cited works as it includes both kinematic information and muscular information. The initial aim of this device is to use EMG and kinematic inputs in a simultaneous but independent way. However, future developments will explore the possibility of sensorial fusion, for instance, by inferring correlations between EMG activity and kinematics. This will allow obtaining a better understanding of how motor control behaves during a rehabilitation task and allow readapting the physical therapy in a more efficient way and improve functional outcomes. ARMIA also offers compatibility with VR and AR applications, as well as virtual on-screen applications. The advantage of this combination with external activities (serious games) was further analyzed in People who have suffered recent brain damage due to stroke, traumatism or any other condition;People with neurodegenerative or neuromuscular diseases of any kind who need regular physical therapy;Elderly people with mobility problems in the upper limb and in need of maintaining physical activity;People affected by post-traumatic or post-surgical complications in arms or hands that require rehabilitation.ARMIA is suitable for a wide range of people with motor limitations, although it could be complemented with cognitive tasks if combined with other tools see . The maiFrom the sensor data, ARMIA software extracts kinematic and muscular parameters to be used with serious games compatible with Virtual Reality (VR) and Augmented Reality (AR) technology. VR is a simulated experience that incorporates different sources of sensory feedback, including auditory, visual or, in some cases, haptic stimuli. In the case of AR, this simulation directly overlays the real world, providing computer-generated perceptual information that complements our view of the surrounding environment. Examples of applications of these technologies to a wide range of motor pathologies can be found in a variety of studies which include approaches with virtual environments ,37,38,39Whac-a-Mole: this game focuses on the rehabilitation of reaching tasks on the transverse plane. The players grab a virtual ball to smash the moles that appear on the screen by reaching their position and hitting them. The score increases on each correct hit. Players have different possibilities for hitting depending on the level of impairment and pathology. One option is to use an external input such as a button to hit the moles or use arm co-contraction to hit the moles at the precise moment. If players have big movement limitations on the hand, the hit can be performed by using the so-called dwell click, which sets the ball to take action automatically when the ball stops moving for a certain amount of time in a certain range to the target. The game will have different difficulty levels that dynamically change depending on players’ performance, or that could be customized by therapists. This difficulty can be based on increasing the number of moles that show up per minute, decreasing the time they are visible, increasing the score required to advance to the next level or including cognitive challenges such as hitting a mole with a particular appearance to obtain extra points.Harvest Truck: this game only takes two control inputs from the wearable: the flexion and extension angle and an action input corresponding to the contraction force level measured on one of the recorded muscles. The players are on the back of a harvest truck traveling on a rural road. On the side of the road, different vegetable boxes of different weights are waiting for pickup. Players must extend the arm to grab the boxes and apply a different amount of contraction force depending on the weight or size of the boxes to be capable of moving them. In this case, in-game difficulty levels change by adding or removing total boxes, increasing box weight or changing the distance to the harvest truck. The score will be computed depending on the number and weight of the collected boxes on each level.By considering these possible control inputs, two different games are being developed for the ARMIA device. These games are particularly focused on planar movements and are a good starting point for the device validation:Future developments for the application of ARMIA will be focused on the evaluation of how motor rehabilitation can be benefited from the use of the proposed gamified activities and how our device can provide the necessary control inputs to allow both clinical rehabilitation and home-based telerehabilitation in an entertaining but effective manner.


The top identified NAS was Staphylococcus epidermidis (28.9%) followed by Staph. chromogenes (27.9%), haemolyticus (15.7%), caprae, and simulans (6.4% each), according to both methods . By MALDI-TOF MS, 13.2% were Staph. microti (2.9%), xylosus (2.0%), equorum, petrasii and warneri (1.5% each), Staph. sciuri , arlettae, capitis, cohnii, lentus (now M. lentus), pseudintermedius, succinus (0.5% each), and 3 isolates (1.5%) were not identified. PCR–RFLP showed 100% AR for Staph. equorum, warneri, arlettae, capitis, and pseudintermedius, 50% for Staph. xylosus, and 0% for the remaining NASM. The top identified streptococcus was Streptococcus uberis (89.5%), followed by Strep. dysgalactiae and parauberis (3.5% each) and by Strep. gallolyticus (1.8%) according to both methods (AR 100%). Only one isolate was identified as a different species by MALDI-TOF MS and PCR–RFLP. In conclusion, MALDI-TOF MS and PCR–RFLP showed a high level of agreement in the identification of the most prevalent NAS and streptococci causing small ruminant mastitis. Therefore, gap PCR–RFLP can represent a good identification alternative when MALDI-TOF MS is not available. Nevertheless, some issues remain for Staph. haemolyticus, minor NAS species including Staph. microti, and species of the novel genus Mammaliicoccus.Staphylococci and streptococci are common causes of intramammary infection in small ruminants, and reliable species identification is crucial for understanding epidemiology and impact on animal health and welfare. We applied MALDI-TOF MS and The online version contains supplementary material available at 10.1186/s13567-022-01102-4. Staphylococcus (Staph.) and Streptococcus (Strep.) [Staphylococcus spp., non-aureus staphylococci (NAS) are the most prevalent and cause mainly subclinical intramammary infection (IMI) in both ewes and goats, leading to significant economic losses and reduced animal welfare [Mammaliicoccus with Mammaliicoccus sciuri as the type species [Mastitis is one of the most common and costly diseases affecting dairy sheep and goats. Therefore, monitoring and maintaining udder health is critical for small ruminant welfare and dairy production yield and quality , 2. Infe(Strep.) . Among S welfare , 5. Rece species . These s species , 7. From species , 8–10. O species .gap gene followed by AluI enzyme digestion for the identification of staphylococci and streptococci isolated from routine diagnosis [In the microbiological laboratory, staphylococci and streptococci isolated from milk samples are mainly identified at the species level by means of biochemical tests or commercial biochemical galleries. However, these methods often fail in correctly identifying bacterial species of veterinary interest, also because they have been optimized for the strains associated with human infections –15. In tiagnosis , 13, 20.gap PCR–RFLP for the species identification of staphylococci and streptococci isolated from the milk of sheep and goats with mastitis. Then, we compared their respective results to identify potential issues and solutions. Bacterial isolates were collected in Sardinia, the region with the largest small ruminant population in Italy.In this study, we applied MALDI-TOF MS and Streptococcus spp. were isolated from sheep and goat milk samples that routinely arrive at the IZSSA laboratories for identification of the IMI agent. Milk samples are accompanied by a form with a checkbox indicating the presence of clinical mastitis/visible milk alterations to be selected by the farm veterinarian. Only isolates derived from samples with these characteristics were included in the study. Milk samples were cultured following the standard procedures provided by the IZSSA. Briefly, 10 µL of milk were seeded in 5% sheep blood agar, incubated at 37 °C, and evaluated at 24 and 48 h. Upon growth of more than one morphologically different bacterial colony, identifications were not performed, and the sample was classified as “mixed bacterial flora”. Colonies were re-isolated in blood agar and examined with Gram stain, catalase, and coagulase tests to discriminate between staphylococci and streptococci. All isolates were stored at −20 °C in Brain Heart Infusion broth  containing 20% glycerol, until further investigation. Only one NASM or streptococcus isolate was selected from each farm for identification by MALDI-TOF MS and PCR–RFLP, for a total of 261 non-duplicate isolates. A geographic map indicating the site of collection of each isolate in the different provinces of Sardinia was created with the software Microsoft Power BI using the farm code of each flock. The geo-referenced coordinates were extracted from the Banca Dati Nazionale (BDN) of the Italian Health Ministry. Additional file From January 2021 to May 2022, 204 NASM and 57 Staphylococcus and Streptococcus species included in this library revision are listed in Additional file Mammaliicoccus species M. sciuri and M. lentus were still reported as Staph. sciuri and Staph. lentus in the library revision available at the time of this study. The following similarity log score thresholds were considered [At the IZSSA, all bacterial isolates were retrieved from the frozen archives, seeded in 5% sheep blood agar, and incubated at 37 °C. All isolates were passaged twice in the solid medium before identification at 24 h of growth. For MALDI-TOF MS analysis, the direct colony transfer protocol was applied. A small amount of an isolated colony was deposited in duplicate wells of disposable target plates using a toothpick, overlaid with 1 µL of α-cyano-4-hydroxycinnamic acid (HCCA) solution in 50% acetonitrile, 47.5% water, 2.5% trifluoroacetic acid , and left to dry. The target plates were prepared at the IZZSA in the afternoon, placed in an empty disposable target container once dry, and transported at room temperature to the animal infectious diseases laboratory at the University of Milan (MILab). In the morning of the following day, i.e. within 24 h as recommended by the MALDI Biotyper™ (MBT) Bruker user manual, the mass spectra were acquired with the MBT Microflex LT/SH MALDI-TOF mass spectrometer  in the positive mode. Each target plate included two spots of Bacterial Test Standard . The obtained spectra were interpreted against the MBT Compass® Library Revision H (2021), covering 3893 species/entries. The nsidered : a log sgap) gene [® Safe DNA gel stain , and visualized under a UV transilluminator.At the IZSSA, genomic DNA was extracted from all 261 isolates and Type/Reference Strains (T/RS) as described previously . Speciesap) gene , 21. TheAluI enzyme . Reaction mixtures were incubated at 37 °C for 15 min and directly loaded on the precast gels. Twenty microliters of digested gap amplifications were loaded in 12% Bis–Tris NuPAGE™ gels (Invitrogen) and then electrophoresed in a vertical gel apparatus. For staphylococci, the following 14 T/RS were used as reference strains for restriction pattern comparison: Staph. epidermidis ATCC 35983, Staph. xylosus ATCC 29971 T, Staph. saprophyticus ATCC 15305 T, Staph. capitis ATCC 27840 T, Staph. haemolyticus ATCC 29970 T, Staph. simulans ATCC 27848 T, Staph. warneri ATCC 27836 T, Staph. arlettae ATCC 43957 T, Staph. chromogenes ATCC 43764 T, Staph. equorum ATCC 43958 T, Staph. caprae ATCC 35538 T, Staph. sciuri ATCC 29062 T, Staph. hiycus ATCC 11249 T, and Staph. intermedius ATCC 29663 T. The gap PCR–RFLP patterns used for Staphylococcus species assignment are illustrated in Additional file Strep. uberis ATCC 700407, Strep. dysgalactiae subsp. dysgalactiae ATCC 43078 T, Strep. dysgalactiae subsp. equisimilis DSM 23147 T, Strep. agalactiae ATCC 13813 T, Strep. gallolyticus subsp. gallolyticus ATCC 49475, Strep. equi subsp. zooepidermicus NCTC 6180, and Strep. suis ATCC 43765. The gap PCR–RFLP patterns used for Streptococcus species assignment are illustrated in Additional file Fifteen microliters of both PCR amplifications were digested in a 30 µL volume containing 10 × FastDigest Green buffer, 0.25 µL of 20 mg/mL acetylated BSA, and 1 µL of FastDigest gap gene amplicons of all isolates with PCR–RFLP profiles different from those of Staphylococcus and Streptococcus T/RS were sequenced at BMR Genomics with the Sanger sequencing option. The nucleotide sequences were compared to sequences in the GenBank database using the Basic Local Alignment Search Tool (BLAST).The gap gene sequencing results, and animal species, were plotted with Microsoft Excel for calculation of agreement ratio (AR), score distribution, and relative percentage values. Plots were generated with Microsoft Excel.All the data related to MALDI-TOF MS results, log scores, PCR–RFLP identification, From January 2021 to May 2022, 204 NASM and 57 streptococci, for a total of 261 isolates, were obtained from sheep and goat milk samples sent to the IZSSA for microbiological analysis with a diagnosis of clinical mastitis/visible milk alterations by the farm veterinarian. Ovine isolates were 246, of which 191 NASM and 55 streptococci. Caprine isolates were 15, of which 13 NASM and 2 streptococci , with log scores ≥ 2.00 in 165 (80.9%) and between 1.70–1.99 in 36 (17.6%). For the remaining 3 isolates (1.5%), log scores were < 1.70 with no identification possible even after a second round of identification with the direct transfer, extended direct transfer, and protein extraction procedures , Staph. chromogenes , Staph. haemolyticus , Staph. caprae , and Staph. simulans , accounting for 85.3% of all NASM isolates. The remaining 13.2% was represented by Staph. microti , Staph. xylosus , Staph. equorum, petrasii, and warneri , Staph. sciuri (now M. sciuri) , and Staph. arlettae, capitis, cohnii, lentus (now M. lentus), pseudintermedius, and succinus .The most frequently identified species were Strep. uberis , followed by Strep. dysgalactiae and parauberis , and by Strep. gallolyticus and suis   with log scores ≥ 2.00 in 53 (93.0%) and between 1.7–1.99 in 4 (7.0%). The most frequently identified species was 00% with gap PCR–RFLP. A total of 187 (91.7%) isolates showed restriction profiles identical to the reference strains, while 17 (8.3%) isolates showed a different PCR–RFLP pattern , devriesei (1), epidermidis (1), haemolyticus (3) hyicus (1), jettensis (1), muscae (1), pseudintermedius (1), pseudoxylosus (1), and simulans (1), while 5 remaining isolates were classified as Staph. muscae (4) and Staph. devriesei (1) by matching with the isolate classified by gap gene sequencing , Staph. chromogenes , and Staph. haemolyticus , followed by Staph. simulans  Staph. caprae , Staph. muscae , Staph. equorum and warneri , Staph. devriesei and xylosus , and Staph. arlettae, capitis, hyicus, jettensis, pseudintermedius, and pseudoxylosus .As a result, the top three identified species were Streptococcus sp. by culture and primary biochemical tests, 53 (93.0%) showed gap PCR–RFLP profiles identical to the reference strains, while 4 (7.0%) did not. Upon gap gene sequencing , Strep. uberis (1), and Strep. ruminantium (1) , followed by Strep. dysgalactiae and parauberis , and by Strep. gallolyticus and ruminantium .Out of the 57 isolates typed as . uberis , and Strrauberis , Strep. gap PCR–RFLP were in general agreement as detailed in Table The species identification results obtained by MALDI-TOF MS and Staph. epidermidis, chromogenes, haemolyticus, caprae, and simulans) the agreement rate (AR) between MALDI-TOF MS and gap PCR–RFLP was 100%. Log scores were mostly ≥ 2.00, although for Staph. haemolyticus only 56.3% of the identifications had log scores ≥ 2.00. The AR between the two identification methods was 100% also for Staph. equorum, warneri, arlettae, capitis, and pseudintermedius. On the other hand, AR was 50% for Staph. xylosus and 0% for Staph. microti, Staph. petrasii, Staph sciuri (now M. sciuri), Staph. cohnii, Staph. lentus (now M. lentus), and Staph. succinus (Table For the 5 most frequently identified NAS (Staph. microti by MALDI-TOF MS and as Staph. muscae (5 isolates) and Staph. simulans (1 isolate) by gap sequencing. Among less frequently identified NASM species, only gap sequencing provided as identifications Staph. muscae, devriesei, hyicus, jettensis, and pseudoxylosus, while only MALDI-TOF MS provided Staph. microti, Staph. petrasii, Staph. sciuri (now M. sciuri), Staph. lentus (now M. lentus), and Staph. succinus.The discordant species identifications between the two approaches are reported in Table gap PCR–RFLP was 98.2% (56 of 57). The AR was 100% for the identification of Strep. uberis, with log scores ≥ 2.00 in 94.1% of cases. The AR was also 100% for Strep. dysgalactiae, parauberis, and gallolyticus. Only one species was identified as Strep suis by MALDI-TOF MS (log score 1.78) and as Strep. ruminantium by gap sequencing  or were not identified by MALDI-TOF MS. In our routine work, Staph. haemolyticus isolated from cow milk is also typically identified with lower scores among the most prevalent Staphylococcus spp. . Staph. haemolyticus may be more problematic to identify because of a higher genomic variability and similarity of marker genes with other species. Wanecka et al. [Staph. haemolyticus isolates had 99.5–100% similarity of the 16S rRNA gene with Staph. petrasii subsp. jettensis (now Staph. petrasii subsp. petrasii), Staph. hominis, Staph. epidermidis, or Staph. devriesei. Also recently, some Staph. haemolyticus have been reclassified as Staph. borealis [gap gene sequencing. On the other hand, the absence of reference spectra for these highly similar minor species in the MALDI-TOF MS database might be the reason for their identification as Staph. haemolyticus with lower log scores.For a et al.  found thborealis . Therefoborealis , as alsoStaph. microti by MALDI-TOF MS could not be identified by gap PCR–RFLP. The gap gene sequence showed the highest identity with Staph. muscae (5 out of 6) and Staph. simulans (1 out of 6). Staph. microti has been described for the first time by Novàkovà et al. in 2010 [Microtus arvalis, the common vole, with Staph. muscae as the nearest relative. Its report as a staphylococcal species associated with mastitis in bovine and bubaline cows is increasing in association with the implementation of MALDI-TOF MS for NAS identification [Staph. rostri, this will require further evaluation, possibly followed by spectrum library integration.All the 6 isolates identified as  in 2010  as an isfication , 32, 33.Staph. petrasii by MALDI-TOF MS with log score ≥ 2.00 was identified as Staph. jettensis upon gap gene sequencing. The species description of Staph. petrasii has been emended, and Staph. jettensis should be reclassified as a novel subspecies within Staph. petrasii for which the name Staphylococcus petrasii subsp. jettensis subsp. nov [Staph. jettensis by gap gene sequencing may have resulted from matching with GenBank sequences that were not updated following taxonomic reclassification.An isolate identified as bsp. nov . As discMammaliicoccus genus, including Staph. sciuri and Staph. lentus, with M. sciuri as the type species [Staph. cohniisubsp. urealyticus as the novel species Staph. urealyticus. The Mammaliicoccus genus is not included in the current release of the Bruker library. Therefore, this should be considered when these species are identified by MALDI-TOF MS with the MBT System and the commercial spectrum library release available at the time of this study.In 2020, five NAS species were proposed to be reassigned to the novel  species . The samStaph. devriesei by gap gene sequencing and PCR–RFLP, one was not identified by MALDI-TOF MS, while the other was identified as Staph. xylosus with a score < 2.00. Moreover, one isolate identified as Staph. hyicus by gap gene sequencing was identified as Staph. sciuri by MALDI-TOF MS with score ≥ 2.00. Staph. devriesei falls in the Staph. haemolyticus group [Staph. sciuri is also reported as belonging to a separate group than Staph. hyicus. Nevertheless, Staph. hyicus can be difficult to differentiate from Staph. agnetis [Concerning the two isolates identified as us group , and Sta agnetis  and the Staph. xylosus by MALDI-TOF MS after a second round of identification with score 1.81. The same isolate was identified as Staph. pseudoxylosus by gap gene sequencing; however, this species was not included in the spectrum library. Analogously, no Strep. ruminantium spectra were present. However, for this latter species a presumptive identification as Strep. suis was provided, with score ≥ 1.7. Strep. ruminantium is indeed a new species of the suis group. Strep. suis includes 35 serotypes, of which 6 have been re-classified to other bacterial species [Strep. suis serotype 33 has been recently classified as a new species, Streptococcus ruminantium [Strep. ruminantium has recently been described [An isolate was identified as  species . Among tinantium , of whicescribed . All othescribed , 40 alsogap gene sequencing to integrate the PCR–RFLP identification approach. If planning such a setup, however, it would be advisable to thoroughly assess the impact of target transportation temperatures and conditions on reliability of the MALDI-TOF MS identification results.As a further consideration, in this work we carried out microbial cultivation and target preparation at the IZSSA in Sassari, Sardinia, and MALDI-TOF MS identification at the University of Milan in Lodi, Lombardy, on the following day. According to the Bruker MBT User Manual, result reliability is maintained if spectra are generated within 24 h from target preparation. Therefore, this opens the possibility that these may be prepared in one laboratory and sent to a shared core facility for identification within the following day, provided that convenient logistic solutions are in place. This approach would enable to contain instrumental costs while centralizing the mass spectrometry instrumentation and the spectrum library, and it might also represent a reasonable alternative to the gap PCR–RFLP provided comparable results. Therefore, gap PCR–RFLP can offer a reliable identification alternative when MALDI-TOF MS is not available, but restriction profiles differing from the validated reference isolates [gap gene sequencing. Concerning MALDI-TOF MS, integrating the spectrum library with small ruminant strains of Staph. haemolyticus as well as of Staph. microti and their closely related species might further improve identification performances and it is advised.In conclusion, our results concerning the species of NASM and streptococci associated with sheep and goat mastitis were in line with those reported as causing small ruminant IMI worldwide , 25. Forisolates , 13, 20 Additional file 1: Geographical distribution of all 261 isolates included in this study.Additional file 2:StaphylococcusandStreptococcusspecies included in the MBT Compass® Library Revision H (2021).Additional file 3:Restriction fragment length polymorphism (RFLP) pattern of PCR products of thegapgene obtained after digestion withAluI and used forStaphylococcusspecies assignment.Additional file 4:Restriction fragment length polymorphism (RFLP) pattern of PCR products of thegapgene obtained after digestion withAluI and used forStreptococcusspecies assignment.Additional file 5:Excel file detailing PCR–RFLP identification,gapgene sequencing information where obtained, MALDI-TOF MS identification, best log score, and animal species.Additional file 6:Genomic sequence of thegapgene and sequence similarity data for staphylococci.Additional file 7:Genomic sequence of thegapgene and sequence similarity data for streptococci.


In Sierra Leone persons with disabilities are at higher risk of living in poverty and have poor access to a fragile healthcare and rehabilitation services. The aim was to explore stakeholders’ perceptions of access to health and rehabilitation services for persons with disabilities in Sierra Leone.Seven focus group discussions, including stakeholders working within the field of disability was conducted.The subthemes were: continuous stigmatisation of persons with disabilities throughout life; long distances and transportation issues to access health and rehabilitation facilities; financial constraints; infrastructural barriers to healthcare and rehabilitation services and healthcare personnel’s negative attitudes and inadequate knowledge towards persons with disabilities; rehabilitation and healthcare facilities lacking materials to provide quality services; lack of specialised services and rehabilitation personnel for complex rehabilitation and the need for continuous education of new and current rehabilitation personnel.Local actors need to take charge and renew efforts made by international organisations by providing trained rehabilitation staff and quality rehabilitation services. Rehabilitation services need to be affordable and transportation costs covered for persons with disabilities to access healthcare and rehabilitation services. Continuous education of the public and health personnel about disability is necessary to reduce negative attitudes towards persons with disabilities. Persons with disabilities (PWD) are at higher risk of living in poverty, being unemployed, and having less access to healthcare compared to the general population . Sierra Sierra Leone has signed the declaration of the Convention on the Rights of Persons Living with Disabilities (CRPD) . ArticleMost PWD in Sierra Leone live in poverty , have poA qualitative study using focus group discussions (FGD) with stakeholders working in the field of disability in Sierra Leone  was appl“Could you please tell us your name, what type of organisation you work for and how long you have been working within health or rehabilitation services? and introductory questions “Could you describe what you work with within health and rehabilitation services?”. Examples of key questions were “In your experience what obstacles do people with disability face in accessing healthcare services? Do you have suggestions on what could be done to improve access for people with disability to health services? If you could change one thing to improve access to rehabilitation services provided for people with disability today, what would you change?” The discussion was summarised by the moderator, followed by an ending question “Is there anything that you feel that we missed and would like to add?”. The questioning route was developed by authors Lina Magnusson (LM) and Victoria Jerwanska (VJ) and validated by staff at Humanity and Inclusion, Sierra Leone. Probing questions were used throughout the FGDs. Ethical approval was obtained from the National Scientific Ethical Review Committee in Sierra Leone  would be the ones like our own ambassadors to talk to the community. We make sure that we incorporate counsellors or the religious leaders in each community in anything we do. They are the ones who reach these people more than us and their voices are stronger than ours”. FGD E, Participant 26, nurse, governmental organisation.Participants disclosed that PWD did not access rehabilitation centres nor use the assistive device due to fear of people seeing and shaming them. They feared ill-treatment from rehabilitation personnel, although assistive devices could give them increased mobility opportunities. Spreading information about rehabilitation services through television and radio was perceived by the participants as a good way of reaching people. As was involving religious and community leaders to raise awareness. Participants discussed religious and community leaders should be regarded as key rehabilitation stakeholders because people trust and listen to them. They could therefore help motivate PWD to seek rehabilitation services.“In Freetown it is much easier to access a vehicle. Thinking about those deep in the provinces, where even to access a vehicle is very difficult. Maybe they use this bike, and the bike rider may not have that patience enough to wait for disabled whose movement is so slow”. FGD C, Participant 12, administration/business, disabled persons organisations.Participants expressed concerns that rehabilitation service coverage was poor and missing in major referral hospitals. Long distances and inadequate transportation to rehabilitation facilities was a challenge experienced by PWD living in both rural and urban areas, which could negatively affect their health condition before arriving at the health facility. Participants described rehabilitation centres lacked funding and transportation to conduct outreach services. Participants described public transport drivers do not stop for PWD due to their slow pace in climbing into the vehicle.“Government is supporting and funding malaria programs and other programs on health. They provided a motorbike for them to do follow up in the community to make sure that things are going as things are supposed to. There are people who have problems in terms of rehabilitation that are living in the rural parts. For the rehabilitation department there are no follow up for people that we are producing devices to. Visit the patient, make sure that the patient is using the devices and how it is using this device. There is no mobility to give support to those activities”. FGD D, Participant 22, certified prosthetics and orthotic technician, governmental organisation.Participants suggested that disability adapted vehicles should be imported. PWD could not afford to pay for long-distance transportation, and they had to pay extra because of being charged for their wheelchair. To ensure access to rehabilitation facilities and assistive devices, participants suggested opening new rehabilitation centres to cover more districts, strengthening rehabilitation services by incorporating community-based rehabilitation (CBR) into peripheral health facilities and training community volunteers in combination with outreach programmes.“PWD, those that are in the deep rural areas, those that are in the villages for them to access a health centre it is very difficult. Even to get their daily living consumption is very difficult. Then to come here to the health centre, where they pay transportation from there to here. When they come, they [healthcare personnel] charge them for the material. Because the service is free but the materials which the personnel is using, they must pay for. Poverty is the thing that hinder them not to come to the health facility”. FGD B, Participant 7, certified rehabilitation therapist, governmental organisation.Participants explained that accessing healthcare and rehabilitation services was expensive due to PWD having to pay for registration, consultation fees and other services like transport within the hospital and drugs even though they should be provided for free. Due to high rates of unemployment, PWD had difficulties raising money to pay for their healthcare. Participants described NGOs used to provide assistive devices free of charge, but that changed when the rehabilitation centres were handed over to the Ministry of Health and Sanitation; PWDs started paying for assistive devices which led to the loss of mobility due to the cost. Some had to sell their belongings to afford an assistive device to meet their necessities. Participants expressed frustration over a lack of materials leading to persons with money being the only one’s able to access assistive devices.Participants suggested decreasing the economic constraints for PWD could increase the utilisation of health and rehabilitation services. Participants called for aid to help the situation for PWD through financial assistance from the Ministry of Social Welfare, Gender and Children’s Affairs or the establishment of a National Disability Trust Fund.“The infrastructure is not disabled-friendly; it is the environment that is making us more disabled. We have nothing here; most hospitals do not have a ramp. So, it means accessibility is a challenge. You have to be dehumanised by [people] carrying you for you to be able to go to some of these hospitals”. FGD F, Participant 33, disability advocate, disabled persons organisations.Participants expressed that accessibility for PWD was not taken into consideration when building public structures. Generally, public buildings, including healthcare and rehabilitation facilities, were not physically accessible for PWD. Participants described limited availability of ramps, escalators, rails and narrow doors caused missed opportunities more broadly in education and employment but also in accessing treatment from healthcare and rehabilitation facilities.“We had an amputee that had a shoulder dislocation, we struggled to get anaesthetics so that they can fix this shoulder because of the pain. It took us two days, begging, begging. Not until one guy who is a student, I spoke to him, he came and helped us, and we fixed the shoulder. But if this is a normal person coming with shoulder dislocation, “Okay, come!” Because they are expecting a huge amount [of money] out of that patient”. FGD B, Participant 27, certified prosthetics and orthotic technician, governmental organisation.Participants disclosed the free healthcare policy for PWD led to healthcare personnel not attending to PWD or treating them last because they could not or should not pay for the healthcare service. Participants expressed that PWD faced more stigmatisation from healthcare personnel than rehabilitation personnel. Healthcare professionals’ ignorance and lack of pride in working with PWD affected PWD’s care.“Lack of training [for healthcare workers] of different types of disabilities is an issue here that at least needs to be recognised. You may not know what to do with it but to understand that something is not right, so that something needs to be done. Just like you would do with clubfeet. You would recognise that a kid has clubfeet and then do something about it. Usually, it is the health workers that have the initial interaction with patients”. FGD A, Participant 2, teacher, non -governmental organisation.Participants further described healthcare personnel displayed negative attitudes regarding sexual and reproductive health rights for PWD, for example, shaming pregnant women with a disability. Persons with hearing impairments lacked interpreters and faced discrimination by staff in healthcare facilities. Participants raised concerns that doctors, midwives, and nurses lacked experience, knowledge and understanding of working with PWD. Doctors and surgeons had a biomedical perspective on health. Still, they lacked knowledge and a holistic perspective which led, for example, to a person being amputated without a referral to rehabilitation services. Participants suggested the training curriculum for doctors, nurses and other healthcare professionals should include early detection and prevention of disability to ensure appropriate care for PWD and to understand the importance of rehabilitation.“I come in the morning for work, and then my duty is to discharge my responsibility. When my client comes, I would like to give him services, but I do not have the materials to produce assistive devices. How can I work for that client?” FGD D, Participant 22, certified prosthetics and orthotic technician, governmental organisation.Participants explained that although the Free Health Care Initiative had been proclaimed, healthcare centres had not gotten the support they needed to provide services free of charge. Participants described necessary healthcare materials and goods were not available. Rehabilitation centres´ machines, and other equipment needed replacements or maintenance to continue to provide assistive devices. Since the government took over services from NGOs rehabilitation centres lacked electricity and running water to produce assistive devices. Rehabilitation professionals got salaries, but the government provided no materials for assistive devices. Participants explained NGOs used to import the needed materials for assistive devices, pay for patient accommodation and provide vocational training for PWD, but this has stopped. Poor, recycled materials lead to PWD paying for low quality assistive devices. Sometimes PWD were asked to provide their own materials to get assistive devices made.“When you go for orthotic device, they will charge you for producing them, but the problem is PWD do not have money. As I said earlier, most of them beg in the streets. Some of them used to work with crutches and orthotic device, but now you find them on wheelchairs, some are even crawling on the ground because there is no money for all these things”. FGD F, Participant 29, disability advocate, non-governmental organisation.Participants expressed frustration seeing PWD lose their mobility as they could not provide or maintain assistive devices as before or knowing that PWD were paying for long-distance transportation to reach rehabilitation centres but would not get the anticipated help.Participants described that importing assistive devices for PWD is tax-free, but there were still challenges. An NGO had trained carpenters and tailors to make rehabilitation equipment that was used across the country. They suggested that industry within the country should make assistive devices and small deposit fees for crutches to facilitate multiple persons use.“How many people do we have that are professionals and can handle eye problems? How many do we really have that can make prostheses for PWD? How many doctors do we really have that can perform spina bifida operations? There should be provisions to ensure Sierra Leoneans to the sector”. FGD G, Participant 35, leadership/policy expert, Governmental organisation,Participants expressed that multiple groups of PWD had complex healthcare and rehabilitation needs. They described rehabilitation services and domestic trained professionals such as occupational therapists, physical therapists, audiologists, and certified prosthetist/orthotist technicians were lacking in most districts. They feared the time they would retire, there would be an increase in the need for rehabilitation services, which would be a problem unless newly qualified professionals were trained.“The problem in Sierra Leone is these multidisciplinary team. We do not have the surgeons, the orthopaedic surgeons. It should be a team that are doing the operation. The physiotherapist should be there, the prosthetist should be there so that they can guide you. The way you are doing the amputation, so that they can affect the way you prepare the prosthesis, but they are not really doing this in Sierra Leone”. FGD C, Participant 14, certified physiotherapist, governmental organisation.Participants described specialised health and rehabilitation wards were lacking to treat persons with cerebral palsy, cerebral malaria, stroke, and spinal cord injuries. Participants expressed hope because a specialised unit on stroke and spinal cord injuries was under development, and improved internet infrastructure could increase the possibility to use more advanced assistive technologies. Participants expressed integrating rehabilitation services into regional hospitals had increased the access to rehabilitation services for a broader group of patients. Participants suggested greater teamwork between healthcare and rehabilitation professionals could lead to improved treatment and rehabilitation outcomes. Participants described that persons with visual impairments lacked assistive devices like canes, braille printers and computers and using health technology to assist PWD was not developed in-country. The lack of sign language interpreters caused communication barriers between healthcare and rehabilitation personnel and persons with hearing impairments. Rehabilitation personnel like prosthetist/orthotist technicians were not currently under a government scheme of service; however, this was under development and awaited validation. A new physiotherapy education had been launched in Sierra Leone, but participants feared that newly trained physiotherapists would leave the country. Participants suggest qualified rehabilitation personnel were needed to stop stigmatisation by sensitising communities and counselling in rural areas. Rehabilitation personnel cared for the services and expressed pride in working in rehabilitation for PWD.“There are people going for all sorts of courses in the medical schools. Some go for nurses, some go for dentistry, but people do not go for rehabilitation because, in the college of medicine, there is no curriculum for that. If we have professionals that could be trained, and the curriculum could be introduced into the educational sector of the country. I think that can enhance the continuous training of rehabilitation service providers in the country”. FGD D, Participant 17, certified occupational therapist, governmental organisation.Participants agreed that strengthening rehabilitation services requires the education of additional rehabilitation professionals. Current rehabilitation personnel needed further training to provide better rehabilitation services. People often go abroad to become rehabilitation professionals as prosthetists and orthotist had no training possibilities domestically. Participants described there were no rehabilitation courses at the college of medicine; however, the first bachelor’s degree in physiotherapy opened in 2019. However, participants were worried that the physiotherapy school would not have enough qualified teachers or rehabilitation personnel to supervise students or the needed equipment to ensure high-quality education.“There is a need for a link, gaining experience from other physiotherapy schools. Some of our physiotherapist went to Tanzania and Kenya. If there is this link with other schools who are practising the modern approach and interventions, we can be capacitated”. FGD A, Participant 1, disability advocate, non-governmental organisation.Participants suggested creating networks with other physiotherapy schools internationally was needed to exchange experiences and increase the quality of education. Furthermore, participants suggested approaching NGOs to fund educational opportunities for rehabilitation subjects.Stakeholders identified that barriers to accessing healthcare and rehabilitation services for PWD were financial constraints, long distances and inaccessible environments to the health and rehabilitation facilities, and negative attitudes of healthcare personnel. Rehabilitation and healthcare facilities lacked specialised services, educated personnel and materials to provide quality services for complex rehabilitation needs. PWD continued to face stigmatisation throughout life, and there was a need to build their capacity for greater inclusion in the Sierra Leonean society. The major findings from the stakeholders perspective correspond with the experiences of persons with physical disabilities accessing and using rehabilitation services in Sierra Leone .Financial constraints for PWD were a barrier to accessing rehabilitation services, and this finding confirmed the results of several previous studies –22. In SOur results indicated long distances and inaccessible environments as barriers to accessing health and rehabilitation services for PWD. In Sierra Leone, 7 out of 10 persons with a disability live in rural communities, and people living in rural areas were poorer than those living in urban areas . PreviouOur results indicated a lack of governance and disjointed health and rehabilitation service delivery in Sierra Leone. Health and rehabilitation service delivery was fragmented and lacked specialised services, educated personnel and materials to provide quality services for basic and complex rehabilitation needs. The urgency to increase the capacity of the health and rehabilitation workforce has been brought to attention both internationally and in Sierra Leone , 23–25. A systematic literature review  includinOne global response in low resource settings to the lack of rehabilitation personnel is CBR. CBR is now widely used across disciplines, mostly in low-income settings, to provide rehabilitation for PWD in their communities by community rehabilitation workers , 29–31. Rehabilitation services faced major constraints with low quality and unavailability of materials, insufficient equipment, and facilities, which led to the poor realisation of rehabilitation services needed by PWD. These findings confirm earlier studies , 26 wherNegative attitudes of healthcare personnel towards PWD in Sierra Leone affects their access to health and rehabilitation services and corresponds to the results of a systematic literature review which including several low and middle-income countries where PWD experienced negative attitudes, lack of communication and lack of knowledge about disability issues from healthcare personnel . A crossPWD continued to face stigmatisation throughout life, and there is a need to build their capacity for greater inclusion in the Sierra Leonean society. These findings correspond with previous studies reporting a low awareness of disability issues among the society and rehabilitation personnel expressing that PWD was marginalised , 26, 34.The strength of this study is the inclusion of seven FGD, 37 participants generating a rich saturated data material. Participants in the FGDs representing a large variation of stakeholders working within the field of disability across several locations in Sierra Leone providing a good national coverage of the workforce within the field, however participants from the Kono district were missing. Limitations in the qualitative study design used has limited transferability  of the rIn conclusion, rehabilitation services need to be free or low cost to be affordable for the PWD in Sierra Leone. To increase access, transportation costs for PWD living in rural areas and far from rehabilitation services need to be covered. To scale up and ensure rehabilitation services increased numbers of locally trained qualified rehabilitation staff is needed. PWD continued to face stigmatisation throughout life, and there is a need to build their capacity and educate the public on disability issues for greater inclusion in society.CBR programmes and workers can facilitate greater inclusion in the society for PWD, especially in remote rural areas. Still, rehabilitation centres with qualified rehabilitation personnel are needed for specialised services to which CBR workers can refer PWD to. Local actors need to take charge and renew efforts made by international organisations by providing trained rehabilitation staff and quality rehabilitation services in several geographical locations and develop a national referral system. In addition, continuous education about disability issues and inclusion is necessary for the public and healthcare personnel to reduce negative attitudes towards PWD.