Ansys Viscoelastic Tissue

Arxiv:2301.04614V1 [Cs.Lg] 11 Jan 2023

Real-time simulation of viscoelastic tissue behavior with

Physics-Guided Deep Learning

Mohammad Karamia,b,∗, Herv´e Lombaertb, David Rivest-H´enaulta aNational Research Council of Canada, 75 de Mortagne Blvd, Boucherville, Qu´ebec, Canada

Abstract

Finite element methods (FEM) are popular approaches for simulation of soft tissues with elastic or viscoelastic behavior. However, their usage in real-time applications, such as in virtual reality surgical training, is limited by computational cost. In this application scenario, which typically involves transportable simulators, the computing hardware severely constrains the size or the level of details of the simulated scene. To address this limitation, data-driven approaches have been suggested to simulate mechanical deformations by learning the mapping rules from FEM generated datasets. Prior data-driven approaches have ignored the physical laws of the underlying engineering problem and have consequently been restricted to simulation cases of simple hyperelastic materials where the temporal variations were effectively ig- nored. However, most surgical training scenarios require more complex hyperelastic models to deal with the viscoelastic properties of tissues.

This type of material exhibits both viscous and elastic behaviors when subjected to external force, requiring the implementation of time-dependant state variables. Herein, we propose a deep learning method for predicting displacement fields of soft tissues with viscoelastic properties. The main contribution of this work is the use of a physics-guided loss function for the optimization of the deep learning model parameters. The proposed deep learning model is based on convolutional (CNN) and recurrent layers (LSTM) to predict spatiotemporal variations. It is augmented with a mass conservation law in the lost function to prevent the generation of physically inconsistent results. The deep learning model is trained on a set of FEM datasets that are generated from a commercially avail- able state-of-the-art numerical neurosurgery simulator. The use of the physics-guided loss function in a deep learning model has led to a better generalization in the prediction of deformations in unseen simulation cases. Moreover, the proposed method achieves a better accuracy over the conventional CNN models, where improvements were observed in unseen tissue from 8% to 30% depending on the magnitude of external forces. It is hoped that the present investigation will help in filling the gap in applying deep learning in virtual reality simulators, hence improving their computational performance (compared to FEM simulations) and ultimately their usefulness.

Keywords:

Physics-guided deep learning, Real-time simulation, Viscoelastic material, Finite element method

1. Introduction

Neurosurgical training is typically obtained in oper- ating rooms under the direct supervision of experienced neurosurgeons, who are severely limited both in available time and in the number of opportunities to practice. Con- sequently, extraclinical surgical simulation is regularly in- tegrated in training curriculum to provide for more devel- opment opportunities. Evidences from the literature tend to demonstrate that specific skills acquired during simu- lation translate to the operating room . Traditionally, surgical simulation use live animal, bench, or cadaveric

Rivest-H´Enault)

models . Recently, virtual reality simulators have be- come a promising alternative method to meet educational requirements [1, 2, 3, 4, 35]. The use of simulation for sur- gical training can have a measurable and significant clini- cal impact by enabling new possibilities in surgical train- ing. In a randomized control trial, demonstrated that using a simulator with metric-based artificial intelligence (AI) tutoring is more effective than receiving remote tutor- ing by a human instructor, or no tutoring while learning simulated brain tumor resections. In this case, it is the realistic simulation system with bimanual haptic feedback that provides the monitoring data that is required by the AI agent, therefore enabling this application. Additionally, the availability of high precision data streams from surgical simulation allows to automatically categorize the level of expertise of the trainee using machine learning algorithms . Continuous expertise monitoring systems can further assess surgical bimanual performance in real-time, which

January 12, 2023

could be leveraged to provide predictive validation during surgical residency training, allowing the early detection of errors and more efficient training. Virtual reality simulators thus enable trainees to practice on a variety of educational scenarios as well as to enable the definition of new training metrics and applications (e.g. remote train- ing).

1.1. Overview Of Virtual Reality Simulators

To provide realistic touch sensations through haptic feedback devices, virtual reality simulators require a real- time simulation of tissue deformation . The behavior of the tissue is critical during neurosurgery and its simula- tion is challenging due to its complex nonlinear viscoelas- tic nature. Finite Element Methods (FEM) are often used numerical techniques for this type of simulation.

How-

ever, the application of FEM in real-time simulation is currently limited by hardware requirements. While sev- eral types of procedure have been successfully simulated, for example, Cranial Microneurosurgery , others, which require a larger field of view or involve stiffer non-rigid material, are much more challenging to implement. Ex- amples of challenging scenarios include femoral nailing in orthopedic or endonasal neurosurgical procedures. If a simulated scene is compute-bound, the designer needs to fall-back on ad-hoc heuristic methods, which might de- grade realism.

At the technical level, one of the main challenges in FEM is a concurrent update of matrix entries, describing the physical state, by several computational threads. This is difficult to implement efficiently in parallel due to the memory access arrangement .

The Following Body Of

work has been proposed to overcome this limitation. A first type of solutions are those based on model- order reductions [7, 8, 9]. The conventional method for constructing a model with comparatively fewer degrees of freedom is the Proper Orthogonal Decomposition (POD).

POD extracts the main displacement patterns, known as modes, that constitute simplified representations of the tissue displacements. Then a linear combination of these modes is used to approximate the final displacement field.

The use of POD for real-time simulation of tissue behavior has been extensively studied [10, 11, 12] and with satisfy- ing results in linear problems. However, this approach fails in the simulation of highly nonlinear problems [13, 14, 15], such as in neurosurgery simulations. This limitation was circumvented by expressing the modes as a sum of sep- arable nonlinear functions, referred to as Proper Gener- alized Decomposition (PGD) .

However, Due To The

implementation of boundary conditions as a new dimen- sion into PGD, this technique is not well generalized and needs to be modified and combined with kernel principal component analysis .

A second alternative for accelerating FEM simulations exploits machine learning approaches . Based on FEM simulation data, machine learning models learn a function that maps an input, such as external forces, to an out- put, such as nodal displacements, without any previous knowledge of the problem. It has been postulated that if a machine learning model is trained on a FEM dataset, it can be subsequently used to predict the outcome of FEM simulations in real time. Such use of machine learn- ing for real-time simulation of tissue behavior has been applied to simulate various organs, including breasts and livers [19, 20]. However in these works the involved machine learning techniques, notably Support Vector Ma- chine (SVM) and random forest, restrict their application to small nodal displacements, as discussed by .

Recently, Convolutional Neural Networks (Cnns), A

class of machine learning methods, have been used to speed up FEM simulations[21, 24, 31]. CNNs are shown to be powerful in extracting complex patterns with the advan- tage of parallelizing well on modern GPUs. Mendizabal et al. have used CNNs, based on a U-Net architecture , for predicting nodal displacements of an elastic liver in real-time. They have also compared the CNN model

With The Pod Method And Have Shown That The Ac-

curacy of a CNN is higher than that of a POD in a com- parable inference time. Pfeifer et al. have also used fully convolutional neural networks for predicting nodal displacements of three different liver models and have ob- tained a reasonable accuracy within 20 ms of inference

Time. Han El Al. Used A Cnn Model In Conjunction

with a classical solver to approach a multiphysics problem in electrosurgery. Despite the wide use of machine learning in computer vision and natural language processing, applications to problems in physics have often met limited success [25, 26].

This issue is partly due to the absence of physical laws in machine learning algorithms. For instance, Jia et al. have highlighted this issue in the prediction of lake temper- ature profiles. The calculation of the lake temperature is rather similar to the calculation of material displacement (our case) since both variables can be obtained from physi- cal principles of momentum/energy conservation. Jia et al.

showed that the predicted temperature profiles from deep learning techniques are not consistent with respect to the energy equation if the physical law of the problem is absent in the loss function. For this reason, previous deep learning techniques (CNNs) [21, 24] that have suggested speeding up FEM simulations were restricted to simula- tion cases of simple hyperelastic materials where the tem- poral variations were effectively ignored. Combining phys- ical laws with deep learning, referred to as physics-guided deep learning, has received considerable attention in recent years. Willard et al. have provided a comprehensive review on this subject.

1.2. Contribution

There is a substantial difference in the effectiveness of virtual reality simulators achieved using machine learn- ing (data-driven) based systems, compared with FEM- simulation systems utilizing only numerical methods. The main benefit of the former consists of their usage in real- time applications while providing similar results to FEM.

Therefore, we addressed this problem in the research re- ported in this paper. This work proposes, for the first time to the authors’ knowledge, a deep learning method for pre- dicting displacement fields of soft tissues with viscoelastic properties.

Overall, our contribution consists of the integration of a mass-conservation law as a new domain-specific loss func- tion into the proposed deep learning model, which makes it possible to prevent the production of physically inconsis- tent results. This is in contrast to the alternative existing machine learning techniques, which are focused exclusively on sample data, and they fail in the simulation of highly nonlinear problems like viscoelastic tissue behavior.

To

show the real application of the proposed method (as part of another contribution), we have used the real data from a virtual reality neurosurgery simulator, NeuroTouch, which is a commercial product. The data is unique and challeng- ing because of the hyperelasticity behavior of the tissue.

We have also demonstrated the faster computation of our proposed method in comparison to the virtual reality sim- ulator results, which are based on FEM technique. It is hoped that the present investigation may also help in fill- ing the gap in applying deep learning in FEM simulations.

1.3. Paper Structure

The paper is organized as follows.

In Section 2, We

present some information about commercial neurosurgi- cal simulators, with particular attention given to the im- plemented FEM technique in NeuroTech. The proposed model is described in Section 3, and the results of the pro- posed model are reported and discussed in Sections 4 and 5. The paper is concluded with Section 6.

2. Background Information

Commercial neurosurgical simulators are based on tis- sue deformation models derived from the fundamentals of continuum mechanics. In such real-time applications, a common strategy is to use simplified numerical FEM models that are less costly to compute than more ana- lytical alternatives. Since one of the motivations for this work is to emulate the output of a FEM-based simula- tor using a data-driven approach (hopefully at a greater computational rate), the main driving equations are pre- sented here. The selected FEM reference software, which will be presented in Sec. 4, uses an explicit time-integration scheme, and tissues are modeled as viscoelastic solids using a quasilinear viscoelastic constitutive model1. The relax-

(1)

where τk is a relaxation time and gk is a relaxation modulus. The elastic part of tissue behavior is modeled as hyperelastic solids of the generalized Neo-Hookean consti- tutive model defined by the following strain energy density

(2)

where C1 is the material constant determined from exper- iments, I1 is the first invariant of the left Cauchy-Green deformation tensor, K is the bulk modulus, and j is the square root of the third invariant of the left Cauchy-Green deformation tensor and a measure of the volumetric defor-

(3)

where µ is the shear modulus, and ν is the Poisson ratio.

3. Method

In this section, data-driven approaches, based on deep neural network architectures, are proposed for speeding up the FEM simulation software. The deep learning models are trained on a set of FEM datasets generated from a commercial-grade simulator. The input to the model is an external force on each node, a tensor of Nx × Ny × Nz × 3, where N is the number of nodes in the three directions of x, y, z (as an example, a brain tumor grid mesh used in this work for real-time simulation has size Nx = 17, Ny = 17, and Nz = 8). The model output is a predicted nodal displacement field with the same shape as the input tensor.

1In mechanics of materials, a constitutive model is a represen- tation of the constitutive laws which are the governing equations describing the behavior of the material.

3

3.1. Existing Neural Network models for tissue displace-

Ment Simulation

U-Net, a neural network architecture based on convo- lutional layers, has already been experimented suc- cessfully for the simulation of simple hyperelastic mate- rials .

However, This Model Is Only Able To Extract

spatial characteristics of the nodal displacement field and ignores the temporal deformation of the viscoelastic ma- terial, which might reduce temporal smoothness and pre- cision. In this work, we propose to alleviate these issues by defining a model that includes an LSTM layer. In ad- dition, terms that are specific to the mechanical nature of the problem are also considered.

Our Proposed Deep Neural Network Model Combines

Convolutional Neural Network (CNN) and Long Short- Term Memory (LSTM) within the architecture shown in Figure 1. LSTM, which is one of the Recurrent Neural Network (RNN) models, is different from CNN models in its consideration of time sequences.

Lstm Contains An

input gate, an output gate, and a forget gate. The inte- gration of these gates makes LSTM a suitable model for time-dependent problems. However, viscoelastic materi- als with complex hyperelastic models are very nonlinear to be modeled with a generic CNN-LSTM model. Herein, the CNN-LSTM model with a Mean Squared Error (MSE) loss function is referred to as ”generic CNN-LSTM” model.

This model is based on the CNN model, whose bottleneck consists of an LSTM layer with 512 nodes. Due to the power of the LSTM in feature extraction, we were able to reduce the number of convolutional operations in the model. The encoding/decoding path has a single convolu- tional operation with 32 filters. The designed parameters of the network architecture are shown in Table 1. It shows the settings and the number of trainable parameters of the entire layers, including convolutional, activation, pooling, LSTM, and upsampling layers.

The Model Optimizer Is

Adam with a learning rate of 0.00001. 3.2.2. Mechanical displacement-specific loss function To improve the modeling of viscoelastic materials, we regularize the deep learning model by an auxiliary task to ensure that the nodal displacement maps are constrained by the constitutive laws. The conventional loss function in regression problems is an MSE between the model pre- dictions, ˆUn,t, and the FEM simulations output, Un,t, for every node and every time step.

To favorably constrain the learning process, a term based on mass conservation law can be considered to ex- tend the loss function. When the model satisfies the mass conservation law, the volume of the tumor at each time step, Vt should always remain constant and equal to the initial volume of undeformed material, Vorigin. Thus, the predicted nodal displacement fields that violate, ∆V = Vt −Vorigin = 0, will be penalized.

One Usual Way Of

penalization is using Rectified Linear Units (ReLU) as ac- tivation functions, but, in this study, it was observed that this function suffers from the vanishing gradient problem.

Thus, to avoid the vanishing gradient inherent from ReLU, a squared difference between Vt and Vorigin, i.e., L2 loss function, has been considered for penalization.

Finally,

the combined loss function, referred to as physics-guided,

(4)

where λ is a hyper-parameter that regulates the balance between LMSE and LP hysics, and it is equal to 0.1. It was observed that decreasing λ to 0.01 will cause the effects of LP hysics to be small, or increasing λ to 1 will deteriorate the learning process. It should be noted that the current FEM simulation has a coarse mesh with compressibility, leading us to have some cases with physical error.

To

circumvent this issue, the physics-guided term, LP hysics, has been applied only to the cases whose volume change is above a threshold value, which is 7% of the material volume. In the rest of this paper, the CNN-LSTM model with the physics-guided custom loss function is referred to as our model.

It is worth mentioning that we also tried a different physical law for implementation into the physical loss func- tion, LP hysics.

Based On Hooke’S Law, The Direction Of

the displacement at each node should be consistent with the direction of the external force at each node. In other words, the cosine angle, cos θ, between the external force vector of each node and the displacement vector of each node should be always zero. Thus, we added a new term, ReLU(1−cosθ), into the LP hysics. However, it was found out this new term does not have a significant effect on the learning process, and thus it has been ignored in the proposed physics-guided loss function.

Once the deep learning model has been trained, it can be deployed for real-time simulation of tissue behavior.

Table 1:

The proposed CNN-LSTM architecture. Note that the merging layer consists of a concatenation of the outputs of two predecessor convolutional layers along the channel dimension.

195

Figure 1: CNN U-Net and CNN-LSTM deep learning architectures. The reference CNN U-Net network (see section 4.1.2) does not include the purple LSTM box. The proposed CNN-LSTM general architecture is similar to that of the CNN network, except for the bottleneck (center layers) which consists of an LSTM layer. The main contribution herein is the use of a physics-guided loss function for the optimization of the model parameters.

4. Results

A virtual reality neurosurgery simulator with haptic feedback2 [1, 2], (Figure 2) was used to illustrate the appli- cability of the proposed method for fast viscoelastic tissue displacement simulation. While capable of real-time simu- lation, the simulation software embedded in the simulator is computationally bounded, which prevents operation on more portable hardware, or, alternatively, the use of big- ger mechanical meshes. This surgical simulator has been used for generating FEM simulation dataset of a brain tu- mor.

The Tumor Mesh Has Overall Dimension Nx = 17,

Ny = 17, and Nz = 8, and consists of 1782 nodes and 1314 hexahedron cells, which leads to an input tensor size of (17, 17, 8, 3). It is subjected to different states of two contact forces. The bottom and the peripheral surfaces of the material are subjected to a zero-displacement bound- ary condition, while its top surfaces are left free. The sim- ulator implements equations (1–3). In (1), τ1 and τ2 are respectively 330s and 11s, and g1 and g2 are respectively 2NeuroTouch, from the National Research Council of Canada, commercially available under the NeuroVR trademark from CAE HealthCare.

0.12 and 0.8. In (2) and (3), C1, ν and µ, are respectively 0.0002 Mpa, 0.0004 Mpa, and 0.42.

4.1. Comparison With Alternative Methods

Herein, our proposed technique has been compared with two different techniques for the prediction of tissue de- formation from the external force: linear regression (as a baseline model) and a previously introduced Convolutional Neural Network (CNN) model. The models have been im- plemented in TensorFlow, and the source code is available at: https://github.com/Mkarami3/UNet_FEM.

4.1.1. Baseline Model

The baseline model is a linear regression model. The linear regression model predicts the nodal displacement field as a linear combination of the input force. This model is not expected to give appealing results but is included to serve as a conceptual lower bound. For complex and nonlinear problems, nonlinear approaches, such as deep learning, are usually required.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.