Purpose: In surgical navigation, pre-operative organ models are presented to surgeons during the intervention to help them in efficiently finding their target. In the case of soft tissue, these models need to be deformed and adapted to the current situation by using intra-operative sensor data. A promising method to realize this are real-time capable biomechanical models. Methods: We train a fully convolutional neural network to estimate a displacement field of all points inside an organ when given only the displacement of a part of the organ's surface. The network trains on entirely synthetic data of random organ-like meshes, which allows us to generate much more data than is otherwise available. The input and output data is discretized into a regular grid, allowing us to fully utilize the capabilities of convolutional operators and to train and infer in a highly parallelized manner. Results: The system is evaluated on in-silico liver models, phantom liver data and human in-vivo breathing data. We test the performance with varying material parameters, organ shapes and amount of visible surface. Even though the network is only trained on synthetic data, it adapts well to the various cases and gives a good estimation of the internal organ displacement. The inference runs at over 50 frames per second. Conclusions: We present a novel method for training a data-driven, real-time capable deformation model. The accuracy is comparable to other registration methods, it adapts very well to previously unseen organs and does not need to be re-trained for every patient. The high inferring speed makes this method useful for many applications such as surgical navigation and real-time simulation.
The idea of using neural networks to estimate the outcome of an FEM simulation is not new. @cite_18 have used a fully connected net to predict the velocity and angle of a tennis ball after hitting a racket. A similar approach has been adopted by @cite_30 for predicting the movement of a tumor during brain surgery, but the displacement of the healthy tissue is not considered. @cite_13 estimate the displacement of a patient's rib cage, but only predict the displacement of surface nodes. In contrast, @cite_14 estimate the displacement of a full liver model using neural networks by superimposing basic deformation modes. These approaches show that neural networks can indeed be trained to estimate soft tissue behavior. However, they work with the data of a single patient, requiring re-training (and in some cases even re-designing) the network for every new patient. Additionally, they use the acting surface forces as input which are very difficult to obtain intra-operatively.
@cite_30 @cite_18 @cite_14 @cite_13
2736507014 1980613493 1490520840 2119640412
Abstract Objectives Accurate reconstruction and visualisation of soft tissue deformation in real time is crucial in image-guided surgery, particularly in augmented reality (AR) applications. Current deformation models are characterised by a trade-off between accuracy and computational speed. We propose an approach to derive a patient-specific deformation model for brain pathologies by combining the results of pre-computed finite element method (FEM) simulations with machine learning algorithms. The models can be computed instantaneously and offer an accuracy comparable to FEM models. Method A brain tumour is used as the subject of the deformation model. Load-driven FEM simulations are performed on a tetrahedral brain mesh afflicted by a tumour. Forces of varying magnitudes, positions, and inclination angles are applied onto the brain's surface. Two machine learning algorithms—artificial neural networks (ANNs) and support vector regression (SVR)—are employed to derive a model that can predict the resulting deformation for each node in the tumour's mesh. Results The tumour deformation can be predicted in real time given relevant information about the geometry of the anatomy and the load, all of which can be measured instantly during a surgical operation. The models can predict the position of the nodes with errors below 0.3 mm, beyond the general threshold of surgical accuracy and suitable for high fidelity AR systems. The SVR models perform better than the ANN's, with positional errors for SVR models reaching under 0.2 mm. Conclusions The results represent an improvement over existing deformation models for real time applications, providing smaller errors and high patient-specificity. The proposed approach addresses the current needs of image-guided surgical systems and has the potential to be employed to model the deformation of any type of soft tissue. Abstract The ability to make decisions based on quantities of interest that depend on variables inferred from measurement finds application in different fields of mechanics and physics. The evaluation of the inferred variables, and hence the quantities of interest, from the measurement typically requires the solution of an inverse problem. For example, in medical imaging the elastic heterogeneity of a tumor and its nonlinear elastic response can be used to distinguish benign tumors from their malignant counterparts. These images of linear and nonlinear elastic parameters of tissue are typically obtained by using a measured displacement field and solving a complex inverse elasticity problem. In this paper we consider circumventing the solution of the inverse problem by using measured displacements as input to a deep convolutional neural network (CNN) and training it to classify tumors on the basis of their elastic heterogeneity and nonlinearity. For a simple, 5-layer CNN trained with 8,000 samples for heterogeneity, and a 4-layer CNN trained with 4,000 samples for nonlinear elasticity we report classification accuracies in the range of 99 . 7 − 99 . 9 . The training and testing data are both obtained from the forward solution of finite element models of samples. We also analyze the weights of the trained model to understand the process through which the network extracts features of elastic moduli from the input displacement images. Finally, we apply the nonlinear elasticity classifier, which is trained entirely using simulated data, to displacement images obtained from ten patients with breast lesions and note that it correctly classifies eight out of ten cases. This application illustrates how data from physics-based models can be used in improving the performance of a data-driven algorithm in data-sparse scenarios. Deformation estimation of elastic object assuming an internal organ is important for the computer navigation of surgery. The aim of this study is to estimate the deformation of an entire three-dimensional elastic object using displacement information of very few observation points. A learning approach with a neural network was introduced to estimate the entire deformation of an object. We applied our method to two elastic objects; a rectangular parallelepiped model, and a human liver model reconstructed from computed tomography data. The average estimation error for the human liver model was 0.041 mm when the object was deformed up to 66.4 mm, from only around 3 observations. These results indicate that the deformation of an entire elastic object can be estimated with an acceptable level of error from limited observations by applying a trained neural network to a new deformation. Surgical planners are used to achieve the optimal outcome for surgery. They are especially desired in procedures where a positive aesthetic outcome is the primary goal, such as the Nuss procedure which is a minimally invasive surgery for correcting pectus excavatum (PE) - a congenital chest wall deformity which is characterized by a deep depression of the sternum. The Nuss procedure consists of placement of a met al bar(s) underneath the sternum, thereby forcibly changing the geometry of the ribcage. Because of the prevalence of PE and the popularity of the Nuss procedure, the demand to perform this surgery is greater than ever. Therefore, a Nuss procedure surgical planner is an invaluable planning tool ensuring an optimal physiological and aesthetic outcome. We propose the development and validation of the Nuss procedure planner. First, a generic model of the ribcage is developed. Then, the computed tomography (CT) data collected from actual patients with PE is used to create a set of patient-specific finite element models (FEM). Based on finite element analyses (FEA) a force-displacement data set is created. This data is used to train an artificial neural network (ANN) to generalize the data set. In order to evaluate the planning process, a methodology which uses an average shape of the chest for comparison with results of the Nuss procedure planner is developed. Haptic feedback and inertial tracking is also implemented. The results show that it is possible to utilize this approximation of the force-displacement model for a Nuss procedure planner and trainer. Presents a novel method for validation of nonrigid medical image registration. This method is based on the simulation of physically plausible, biomechanical tissue deformations using finite-element methods. Applying a range of displacements to finite-element models of different patient anatomies generates model solutions which simulate gold standard deformations. From these solutions, deformed images are generated with a range of deformations typical of those likely to occur in vivo. The registration accuracy with respect to the finite-element simulations is quantified by co-registering the deformed images with the original images and comparing the recovered voxel displacements with the biomechanically simulated ones. The functionality of the validation method is demonstrated for a previously described nonrigid image registration technique based on free-form deformations using B-splines and normalized mutual information as a voxel similarity measure, with an application to contrast-enhanced magnetic resonance mammography image pairs. The exemplar nonrigid registration technique is shown to be of subvoxel accuracy on average for this particular application. The validation method presented here is an important step toward more generic simulations of biomechanically plausible tissue deformations and quantification of tissue motion recovery using nonrigid image registration. It will provide a basis for improving and comparing different nonrigid registration techniques for a diversity of medical applications, such as intrasubject tissue deformation or motion correction in the brain, liver or heart. In this paper, an approach for 'virtual' real-time deformation of complex structures is developed. The approach combines the finite element method and Neural Network calculations to allow the user to perform interactive shape changes and view the resultant deformation changes in a virtual environment.A tennis racket and ball are used to illustrate the capability of the proposed method.Because real time deformation simulation is a time consuming repeated analyses, the neural networks are employed in this investigation as numerical devices for substituting the finite element code needed for the tennis racket and ball deformation.The input data for the artificial neural network are a set of parameters generated randomly (ball impact velocity, impact angle and the zone impact). The output data are the deformation of the ball strings and the impact force-feedback value.The work contribute toward the development of real virtual simulator which uses a haptic device that "feels" the feedback force generated by the deformation. This paper presents a new method for simulating the deformation of organ models by using a neural network. The proposed method is based on the idea proposed by [2] that a deformed model can be estimated from the superposition of basic deformation modes. The neural network finds a relationship between external forces and the models deformed by the forces. The experimental results show that the trained network can achieve a real-time simulation while keeping the acceptable accuracy compared with the nonlinear FEM computation. Machine learning (ML) to model the liver biomechanical behaviour during breathing.ML is much faster than the popular FEM, allowing real-time soft tissue modelling.Modelling scheme able to predict deformation for a new load and a new liver.ML regression models were used: three tree-based methods and two simpler ones.Good prediction performance was obtained: all samples with an error under 1źmm. Progress in biomechanical modelling of human soft tissue is the basis for the development of new clinical applications capable of improving the diagnosis and treatment of some diseases (e.g. cancer), as well as the surgical planning and guidance of some interventions. The finite element method (FEM) is one of the most popular techniques used to predict the deformation of the human soft tissue due to its high accuracy. However, FEM has an associated high computational cost, which makes it difficult its integration in real-time computer-aided surgery systems. An alternative for simulating the mechanical behaviour of human organs in real time comes from the use of machine learning (ML) techniques, which are much faster than FEM. This paper assesses the feasibility of ML methods for modelling the biomechanical behaviour of the human liver during the breathing process, which is crucial for guiding surgeons during interventions where it is critical to track this deformation (e.g. some specific kind of biopsies) or for the accurate application of radiotherapy dose to liver tumours. For this purpose, different ML regression models were investigated, including three tree-based methods (decision trees, random forests and extremely randomised trees) and other two simpler regression techniques (dummy model and linear regression). In order to build and validate the ML models, a labelled data set was constructed from modelling the deformation of eight ex-vivo human livers using FEM. The best prediction performance was obtained using extremely randomised trees, with a mean error of 0.07źmm and all the samples with an error under 1źmm. The achieved results lay the foundation for the future development of some real-time software capable of simulating the human liver deformation during the breathing process during clinical interventions. Abstract This work presents a data-driven method to simulate, in real-time, the biomechanical behavior of the breast tissues in some image-guided interventions such as biopsies or radiotherapy dose delivery as well as to speed up multimodal registration algorithms. Ten real breasts were used for this work. Their deformation due to the displacement of two compression plates was simulated off-line using the finite element (FE) method. Three machine learning models were trained with the data from those simulations. Then, they were used to predict in real-time the deformation of the breast tissues during the compression. The models were a decision tree and two tree-based ensemble methods (extremely randomized trees and random forest). Two different experimental setups were designed to validate and study the performance of these models under different conditions. The mean 3D Euclidean distance between nodes predicted by the models and those extracted from the FE simulations was calculated to assess the performance of the models in the validation set. The experiments proved that extremely randomized trees performed better than the other two models. The mean error committed by the three models in the prediction of the nodal displacements was under 2 mm, a threshold usually set for clinical applications. The time needed for breast compression prediction is sufficiently short to allow its use in real-time ( 0.2 s). Hitting and batting tasks, such as tennis forehands, ping-pong strokes, or baseball batting, depend on predictions where the ball can be intercepted and how it can properly be returned to the opponent. These predictions get more accurate over time, hence the behaviors need to be continuously modified. As a result, movement templates with a learned global shape need to be adapted during the execution so that the racket reaches a target position and velocity that will return the ball over to the other side of the net or court. It requires altering learned movements to hit a varying target with the necessary velocity at a specific instant in time. Such a task cannot be incorporated straightforwardly in most movement representations suitable for learning. For example, the standard formulation of the dynamical system based motor primitives (introduced by [1]) does not satisfy this property despite their flexibility which has allowed learning tasks ranging from locomotion to kendama. In order to fulfill this requirement, we reformulate the Ijspeert framework to incorporate the possibility of specifying a desired hitting point and a desired hitting velocity while maintaining all advantages of the original formulation. We show that the proposed movement template formulation works well in two scenarios, i.e., for hitting a ball on a string with a table tennis racket at a specified velocity and for returning balls launched by a ball gun successfully over the net using forehand movements. All experiments were carried out on a Barrett WAM using a four camera vision system. Efficient and informative visualization of surfaces with uncertainties is an important topic with many applications in science and engineering. In these applications, the correct course of action may depend not only on the location of a boundary, but on the precision with which that location is known. Examples include environmental pollution borderline detection, oil basin edge characterization, or discrimination between cancerous and healthy tissue in medicine. We present a method for producing visualizations of surfaces with uncertainties using points as display primitives. Our approach is to render the surface as a collection of points and to displace each point from its original location along the surface normal by an amount proportional to the uncertainty at that point. This approach can be used in combination with other techniques such as pseudocoloring to produce efficient and revealing visualizations. The basic approach is sufficiently flexible to allow natural extensions; we show incorporation of expressive modulation of opacity, change of the stroke primitive, and addition of an underlying polygonal model. The method is used to visualize real and simulated tumor formations with uncertainty of tumor boundaries. The point-based technique is compared to pseudocoloring for a position estimation task in a preliminary user study. Understanding the 3D structure of a scene is of vital importance, when it comes to developing fully autonomous robots. To this end, we present a novel deep learning based framework that estimates depth, surface normals and surface curvature by only using a single RGB image. To the best of our knowledge this is the first work to estimate surface curvature from colour using a machine learning approach. Additionally, we demonstrate that by tuning the network to infer well designed features, such as surface curvature, we can achieve improved performance at estimating depth and normals. This indicates that network guidance is still a useful aspect of designing and training a neural network. We run extensive experiments where the network is trained to infer different tasks while the model capacity is kept constant resulting in different feature maps based on the tasks at hand. We outperform the previous state-of-the-art benchmarks which jointly estimate depths and surface normals while predicting surface curvature in parallel. Identifying historical records of patients who are similar to the new patient could help to retrieve similar reference cases for predicting the clinical outcome of the new patient. Amongst different potential applications, this study illustrates use of patient similarity in predicting survival of patients suffering from hepatocellular carcinoma (HCC) treated with locoregional chemotherapy. This study used 14 similarity measures derived from relevant clinical and imaging parameters to classify the HCC patient pairs into two classes, namely the difference between their survival time being longer or no longer than 12 months. Furthermore, this paper proposes and presents a patient similarity algorithm for the classification, named SimSVM. With the 14 similarity measures as input, SimSVM outputs the predicted class and the degree of similarity or dissimilarity. A dataset was collected from 30 patients, forming 300 and 135 patient pairs as training and test datasets respectively. The trained SimSVM with linear kernel gave the best accuracy (66.7 ), sensitivity (64.8 ) and specificity (67.9 ) on the test dataset. This paper describes an approach to identify plastic deformation and failure properties of ductile materials. The experimental method of the small punch test is used to determine the material response under loading. The resulting load displacement curve is transferred to a neural network, which was trained using load displacement curves generated by finite element simulations of the small punch test and the corresponding material parameters. The simulated material behavior of the specimen is based on the ductile elastoplastic damage theory of Gurson, Tvergaard and Needleman. During a training process the neural network generates an approximated function for the inverse problem relating the material parameters to the shape of the load displacement curve of the small punch test. This technique was tested for three different materials (ductile steels). The identified parameters are verified by testing and simulating notched tensile specimens. Display Omitted Survival prediction of HCC patients is modeled in a real and high complex scenario.Its complexity is due to a small dataset with heterogeneous and missing input data.We propose a new cluster-based oversampling method to overcome these drawbacks.Neural networks and logistic regression are applied in the classification stage.Experimental results show that our method outperforms previous existing approaches. Liver cancer is the sixth most frequently diagnosed cancer and, particularly, Hepatocellular Carcinoma (HCC) represents more than 90 of primary liver cancers. Clinicians assess each patients treatment on the basis of evidence-based medicine, which may not always apply to a specific patient, given the biological variability among individuals. Over the years, and for the particular case of Hepatocellular Carcinoma, some research studies have been developing strategies for assisting clinicians in decision making, using computational methods (e.g. machine learning techniques) to extract knowledge from the clinical data. However, these studies have some limitations that have not yet been addressed: some do not focus entirely on Hepatocellular Carcinoma patients, others have strict application boundaries, and none considers the heterogeneity between patients nor the presence of missing data, a common drawback in healthcare contexts. In this work, a real complex Hepatocellular Carcinoma database composed of heterogeneous clinical features is studied. We propose a new cluster-based oversampling approach robust to small and imbalanced datasets, which accounts for the heterogeneity of patients with Hepatocellular Carcinoma. The preprocessing procedures of this work are based on data imputation considering appropriate distance metrics for both heterogeneous and missing data (HEOM) and clustering studies to assess the underlying patient groups in the studied dataset (K-means). The final approach is applied in order to diminish the impact of underlying patient profiles with reduced sizes on survival prediction. It is based on K-means clustering and the SMOTE algorithm to build a representative dataset and use it as training example for different machine learning procedures (logistic regression and neural networks). The results are evaluated in terms of survival prediction and compared across baseline approaches that do not consider clustering and or oversampling using the Friedman rank test. Our proposed methodology coupled with neural networks outperformed all others, suggesting an improvement over the classical approaches currently used in Hepatocellular Carcinoma prediction models. A significant weakness of most current deep Convolutional Neural Networks is the need to train them using vast amounts of manually labelled data. In this work we propose a unsupervised framework to learn a deep convolutional neural network for single view depth prediction, without requiring a pre-training stage or annotated ground-truth depths. We achieve this by training the network in a manner analogous to an autoencoder. At training time we consider a pair of images, source and target, with small, known camera motion between the two such as a stereo pair. We train the convolutional encoder for the task of predicting the depth map for the source image. To do so, we explicitly generate an inverse warp of the target image using the predicted depth and known inter-view displacement, to reconstruct the source image; the photometric error in the reconstruction is the reconstruction loss for the encoder. The acquisition of this training data is considerably simpler than for equivalent systems, requiring no manual annotation, nor calibration of depth sensor to camera. We show that our network trained on less than half of the KITTI dataset gives comparable performance to that of the state-of-the-art supervised methods for single view depth estimation. The prediction of clinical outcome of patients after breast cancer surgery plays an important role in medical tasks such as diagnosis and treatment planning. Different prognostic factors for breast cancer outcome appear to be significant predictors for overall survival, but probably form part of a bigger picture comprising many factors. Survival estimations are currently performed by clinicians using the statistical techniques of survival analysis. In this sense, artificial neural networks are shown to be a powerful tool for analysing datasets where there are complicated non-linear interactions between the input data and the information to be predicted. This paper presents a decision support tool for the prognosis of breast cancer relapse that combines a novel algorithm TDIDT (control of induction by sample division method, CIDIM), to select the most relevant prognostic factors for the accurate prognosis of breast cancer, with a system composed of different neural networks topologies that takes as input the selected variables in order for it to reach good correct classification probability. In addition, a new method for the estimate of Bayes' optimal error using the neural network paradigm is proposed. Clinical-pathological data were obtained from the Medical Oncology Service of the Hospital [email protected]?nico Universitario of Malaga, Spain. The results show that the proposed system is an useful tool to be used by clinicians to search through large datasets seeking subtle patterns in prognostic factors, and that may further assist the selection of appropriate adjuvant treatments for the individual patient. In this paper we propose a method to solve nonrigid image registration through a learning approach, instead of via iterative optimization of a predefined dissimilarity metric. We design a Convolutional Neural Network (CNN) architecture that, in contrast to all other work, directly estimates the displacement vector field (DVF) from a pair of input images. The proposed RegNet is trained using a large set of artificially generated DVFs, does not explicitly define a dissimilarity metric, and integrates image content at multiple scales to equip the network with contextual information. At testing time nonrigid registration is performed in a single shot, in contrast to current iterative methods. We tested RegNet on 3D chest CT follow-up data. The results show that the accuracy of RegNet is on par with a conventional B-spline registration, for anatomy within the capture range. Training RegNet with artificially generated DVFs is therefore a promising approach for obtaining good results on real clinical data, thereby greatly simplifying the training problem. Deformable image registration can therefore be successfully casted as a learning problem. This paper develops a procedure for quantifying movement sequences in terms of move length and turning angle probability distributions. By assuming that movement is a correlated random walk, we derive a formula that relates expected square displacements to the number of consecutive moves. We show this displacement formula can be used to highlight the consequences of different searching behaviors (i.e. different probability distributions of turning angles or move lengths). Observations of Pieris rapae (cabbage white butterfly) flight and Battus philenor (pipe-vine swallowtail) crawling are analyzed as a correlated random walk. The formula that we derive aptly predicts that net displacements of ovipositing cabbage white butterflies. In other circumstances, however, net displacements are not well-described by our correlated random walk formula; in these examples movement must represent a more complicated process than a simple correlated random walk. We suggest that progress might be made by analyzing these more complicated cases in terms of higher order markov processes. The impressive capabilities of the mammalian brain—ranging from perception, pattern recognition and memory formation to decision making and motor activity control—have inspired their re-creation in a wide range of artificial intelligence systems for applications such as face recognition, anomaly detection, medical diagnosis and robotic vehicle control. Yet before neuron-based brains evolved, complex biomolecular circuits provided individual cells with the ‘intelligent’ behaviour required for survival. However, the study of how molecules can ‘think’ has not produced an equal variety of computational models and applications of artificial chemical systems. Although biomolecular systems have been hypothesized to carry out neural-network-like computations in vivo and the synthesis of artificial chemical analogues has been proposed theoretically, experimental work has so far fallen short of fully implementing even a single neuron. Here, building on the richness of DNA computing and strand displacement circuitry, we show how molecular systems can exhibit autonomous brain-like behaviours. Using a simple DNA gate architecture that allows experimental scale-up of multilayer digital circuits, we systematically transform arbitrary linear threshold circuits (an artificial neural network model) into DNA strand displacement cascades that function as small neural networks. Our approach even allows us to implement a Hopfield associative memory with four fully connected artificial neurons that, after training in silico, remembers four single-stranded DNA patterns and recalls the most similar one when presented with an incomplete pattern. Our results suggest that DNA strand displacement cascades could be used to endow autonomous chemical systems with the capability of recognizing patterns of molecular events, making decisions and responding to the environment.