Simulia Robotic Surgery Simulation

Analysis of Executional and Procedural Errors in Dry-lab Robotic

Surgery Experiments

Kay Hutchinson*1 · Zongyu Li*1 · Leigh A. Cantrell2 ·

Abstract

Background Analyzing kinematic and video data can help identify potentially erroneous motions that lead to sub-optimal surgeon performance and safety-critical events in robot-assisted surgery. Methods We develop a rubric for identifying task and gesture-specific Executional and Procedural errors and evaluate dry-lab demonstrations of Suturing and Needle Passing tasks from the JIGSAWS dataset. We characterize erroneous parts of demonstrations by labeling video data, and use distri- bution similarity analysis and trajectory averaging on kinematic data to identify parameters that distinguish erroneous gestures.

Results Executional error frequency varies by task and gesture, and correlates with skill level. Some predominant error modes in each gesture are distinguishable by analyzing error-specific kinematic parameters. Procedural errors could lead to lower performance scores and increased demonstration times but also depend on surgical style.

Conclusions This study provides insights into context-dependent errors that can be used to design automated error detection mechanisms and improve training and skill assessment.

1 Introduction

With advances in sensing and computing technology, artificial intelligence, and data science, the next generation of Robot-Assisted Surgery (RAS) systems is envisioned to benefit from new capabilities for context-specific monitoring and virtual coaching during simulation training as well as decision support and cognitive assistance during actual surgery to improve safety, efficiency, and quality of care . State-of-the-art RAS systems and simulators are designed with data logging mechanisms to collect system logs, kinematics, and video data from surgical procedures. The recorded data has been mostly used for offline surgical skill evaluation , with the aim of improving surgeons’ performance and making evaluations objective and scalable.

Current methods for objective assessment of robotic technical skills can be classified into two general categories: manual assessment and automated assessment. Manual skill evaluation is usually performed globally, assessing performance over an entire demonstration using frameworks such as OSATS (Objective Structured Assessment of Technical Skills) , GOALS (Global Operative As- sessment of Laparoscopic Skills) , GEARS , and R-OSATS . However, manual assessment methods are subjective, cognitively demanding, and prone to errors . In response, automated assessment methods utilizing kinematic, video, and system event data are being developed to provide objective and quantitative metrics and , and explainable feedback . Automated methods also allow the subdivision of demonstrations into subtasks or gestures, and to base perfor- mance assessment and technical skill evaluation on the quality and/or sequence of these components * Co-first-authors: Contributed equally to the paper.

2

Hutchinson et al. as proposed in , , and . Further, some gestures are more indicative of skill level than others .

The metrics used for surgical skill assessment can be classified into three broad categories of: i) efficiency (e.g., path length, completion time), ii) safety (e.g., instrument collisions , instruments out of view, excessive force, needle drops, tissue damage ), and iii) task/procedure specific metrics (e.g., task outcome metrics, camera movement, energy activation ).

While most previous works focused on skill evaluation for distinguishing between expertise levels, less attention has been paid to identifying specific erroneous surgical motions that contribute to sub-optimal performance and potential safety-critical events. The closest related works are and which proposed objective gesture-based checklists for laparoscopic and robot-assisted suturing.

Others have proposed general and custom rubrics for evaluation of human errors and technical errors in laparoscopic surgery. Related works on errors in RAS mainly focused on analyzing adverse events and system malfunctions as reported by the surgical teams and institutions .

Augmenting RAS systems and simulators with mechanisms for monitoring the progress of surgical tasks and providing early and context-specific feedback to surgeons on potentially sub-optimal or unsafe motions could help improve performance scores in training and prevent safety-critical events in actual surgery .

In this study, we take a step towards developing automated safety monitoring mechanisms for RAS by defining a rubric for identifying task and gesture-specific errors based on video and kinematic data. We use this rubric to analyze recorded dry-lab demonstrations of two common tasks (Suturing and Needle Passing) performed on the da Vinci Surgical System (dVSS). We focus on identifying which parts of a trajectory (spanning one or more gestures) are potentially erroneous (sub-optimal) versus error-free (optimal). We then characterize the erroneous trajectories by identifying the most common types of errors for each task and gesture, and the kinematic parameters and surgeon- specific signatures that distinguish between optimal and sub-optimal performance. The results from this study can aid in designing more efficient training modules, curricula, and simulation tools that reinforce optimal performance by providing detailed, quantitative, and context-specific feedback to surgeons. In summary, we propose a novel framework for objective evaluation of RAS procedures

With The Following Key Contributions1:

– A task and gesture-specific rubric for identification of executional and procedural errors using data collected from real or simulated surgical demonstrations. – A set of executional error labels based on manual annotation of video data to augment the Suturing and Needle Passing tasks of the JIGSAWS dataset .

– Quantitative analysis methods to characterize gesture-specific Executional and task-specific Pro- cedural errors using pre-collected kinematic data and gesture labels. – Insights on the types, frequencies, and durations of Executional and Procedural errors across tasks and gestures and their correlations with skill levels which can provide a basis for the design of automated error detection mechanisms.

2 Methods

Sources of errors in RAS are diverse and domain-specific, including faults in the robotic system software and hardware, or human errors . In this study, we focus on errors in the execution of procedures that can be observed in video recordings and detected in kinematic data. Surgical procedures follow the hierarchy of levels defined in which provides context for actions during the operation, as shown in Figure 1. A surgical operation can involve multiple procedures which are divided into steps. Each step is subdivided into tasks comprised of gestures (also called sub-tasks or surgemes) which are made of motions such as moving an instrument or closing the graspers. Errors can occur at any level of this hierarchy and can propagate and cause errors at other 1 Our labels, code, and data are available at https://github.com/UVA-DSA/ExecProc_Error_Analysis.

Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments

Suturing

Fig. 1: Surgical hierarchy (adopted from ) for an example urological procedure of partial nephrec- tomy (based on and ) along with example gesture-specific Executional and Procedural Errors levels. We specifically focus on studying the quality of the task demonstrations at the gesture level

To Answer The Following Research Questions:

RQ1: Which tasks and gestures are most prone to errors? RQ2: Are there common error modes or patterns across gestures and tasks? RQ3: Are erroneous gestures distinguishable from normal gestures? RQ4: What kinematic parameters can be used to distinguish between normal and erroneous ges-

Tures?

RQ5: Do errors impact the duration of the trajectory? RQ6: Are there any correlations between errors and surgical skill levels? 2.1 Rubric for Objective Assessment of Errors in Robotic Surgery Our goal is to define a rubric for identification of errors based on video and/or kinematic data, which can also be used for automated error detection using quantitative measures such as instrument position, amount of force, traveling distance, and system events. We adopt a previous categorization of human errors in laparoscopic surgery from and define two types of errors in our rubric: Proce- dural errors and Executional errors. Procedural errors involve “the omission or re-arrangement of correctly undertaken steps within the procedure,” while Executional errors are “the failure of a specific motor task within the procedure.” Technical errors are the “failure of a planned action to achieve a goal”, including inadequate (too much/too little) use of force or distance, inadequate visu- alization and wrong orientation of instruments or dissection plane , and are considered a subtype of Executional errors that can be quantified with thresholds.

In order to generalize these definitions to different procedures and tasks, we define Executional and Procedural errors at the gesture level. More specifically, we define a set of Executional error modes for each gesture as listed in the rubric in Table 1. Some errors are gesture-specific such as “Needle orientation” which is only defined for G4 and G8 as those gestures specifically manipulate

4

Hutchinson et al.

61

Table 1: Gesture-specific Executional errors for Suturing and Needle Passing in the JIGSAWS dataset. Example videos for each error mode can be found at https://github.com/UVA-DSA/ ExecProc_Error_Analysis.

the needle in preparation for positioning the needle (G2) and throwing the next suture (G3), as shown in the grammar graph of Figure 1 (adopted from ). The standard acceptable practice for those gestures is to hold the needle in the grasper 1/2 to 2/3 of the way from the tip of the needle and with the needle perpendicular to the jaws of the grasper . Other gestures that do not purposely alter the orientation of the needle in the grasper cannot have this error mode. For G3, the definition of a “Multiple attempts” error also includes “Not moving along the curve” of the needle (from ) since these two errors are very difficult to distinguish and often happen simultaneously. Other error modes, including “Multiple attempts”, “Needle drop”, and “Out of view”, could occur at any time during a task and are considered for every gesture.

We define Procedural errors as any deviation in the sequence of gestures performed in a demon- stration from the standard accepted gesture sequences defined for that task and shown in the gram- mar graphs in Figures 1 and 2. defined several sub-categories for Procedural errors, including adding an unexpected step, skipping a step, out of order transition, and repetition of steps. These subcategories are included in our analysis of Procedural errors as discussed in Section 2.4.

2.2 Jigsaws Dataset

The JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) is a publicly available dataset, collected using the Research API for the da Vinci Surgical System (dVSS) from eight sur- geons of varying skill levels performing three dry-lab surgical tasks: Suturing, Knot Tying, and Needle Passing. These tasks are among the standard modules in most surgical skills training curricula.

Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments

Transcript

Fig. 2: Overall Methodology for Analysis of Executional and Procedural Errors The JIGSAWS dataset includes kinematic and video data from up to 39 demonstrations (or trials) per task along with manually annotated gesture transcripts (indicating the sequence of gestures, with the beginning and end of each gesture and its type) and surgical skill levels for each demonstration. The vocabulary of surgical gestures used for labeling is shown in Table 1. Surgical skills were characterized using both self-proclaimed expertise levels and Global Rating Scale (GRS) score for each demonstration. Self-proclaimed (SP) expertise levels were based on the number of hours of robotic surgical experience, divided into: SP-Expert (>100 hrs), SP-Intermediate (10- 100 hrs), and SP-Novice (<10 hrs). GRS scores were given using a modified Objective Structured Assessments of Technical Skills (OSATS) approach based on 6 elements (on a rating-scale of 1-5 per element): Respect for tissue, Suture/needle handling, Time and motion, Flow of operation, Overall performance and Quality of final product . We also classified the demonstrations into three groups based on the GRS scores: GRS-Novice (0≤GRS ≤9), GRS-Intermediate (10≤GRS ≤19), and GRS-Expert (20≤GRS ≤30).

Figure 2 shows our overall pipeline for the analysis of Executional and Procedural errors in the JIGSAWS dataset. Due to the limited number of demonstrations for the Knot Tying task in the dataset, our analysis only focused on Suturing and Needle Passing.

6

Hutchinson et al.

54-56

Right PSM1 tool tip rotational velocity (α’ β’ γ’)

L Grip Ang

Table 2: Kinematic variables in the JIGSAWS dataset (adopted from )

2.3 Executional Error Analysis

Kinematic and video data for each trial were first segmented into gestures based on the gesture transcript annotations. The video clip for each gesture was then reviewed and labeled by two to three independent annotators (with experience in robotic surgery and/or suturing) as normal or erroneous for each error mode. Final labels for each error mode were obtained by taking the consensus among annotators. A gesture example that exhibited one or more errors was marked as erroneous, otherwise, it was labeled as normal. We then proceeded with the analysis of the patient-side manipulator (PSM) kinematic data corresponding to each gesture for all the normal and erroneous demonstrations of each task.

2.3.1 Dynamic Time Warping

We used Dynamic Time Warping (DTW) to measure the similarity between normal and erroneous trajectories for each gesture. DTW is an effective method for aligning two temporal sequences, independent of the non-linear variations in time, by minimizing the Euclidean distance between the two signals. In our analysis, we performed independent DTW on each variable before summing the returned distances for each parameter listed in Table 2. We found no significant difference between this method and dependent DTW where all variables in each parameter group were warped together yielding a single distance instead of a sum of distances (similar observations were made in ).

DTW was performed on every combination of two example trajectories for each gesture. From this, we obtained comparisons of normal examples to other normal examples (“Nor-Nor”) and comparisons of erroneous examples to normal examples (“Err-Nor”). The DTW distance samples represented a distribution of distances for the “Nor-Nor” and “Err-Nor” subsets as shown in the histogram of Figure 2. This resulted in two sets of distance samples for each parameter, each representing a DTW distribution for a comparison subset.

2.3.2 Kullback-Liebler Divergence

Kullback-Liebler (KL) Divergence, also called relative entropy, is a non-symmetric measure of the difference between two probability distributions. The KL Divergence between two identical distri- butions is zero. As shown in Equation 1, KL Divergence was used to compare the “Err-Nor” and “Nor-Nor” DTW distance distributions for each gesture to determine which parameters had a sig- nificant difference between the two distributions.

(1)

Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments

1 Input:

- A grammar graph G(V, E) for a surgical task which is a digraph with each vertex in V representing the entry point START or one of the common gesture types Gi in the task, and each edge(Gi, Gj) ∈E representing a common transition between gestures Gi and Gj.

- A set of m task transcripts T = {T1, T2, ..., Tm}, where Tk ∈T is an ordered sequence of n gestures

Output:

- A list of erroneous gesture transitions error seq for each transcript

2.3.3 Trajectory Averaging

We examined the kinematic data for important parameters to verify differences between normal and erroneous gestures using a method based on . Each signal was time-normalized by downsampling the signal by 3 (keeping only every third sample) and then linearly interpolated to stretch it to the average duration of the normal or erroneous gesture examples of that task (supported by our analysis of gesture durations in Section 3.1.4). Then, fuzzy c-means clustering was performed on each variable and its normalized time index to obtain the average normal and erroneous trajectories (represented by 15 cluster centers), shown with blue (normal) and red (erroneous) dots in Figure 4b.

2.4 Procedural Error Analysis

Previous works proposed modeling the standard acceptable gesture sequences for a task using a grammar graph that shows the relationship, order, and flow of gestures , . . The grammar graph of a task is a digraph with each vertex representing the set of gestures for the task and each edge representing a common transition between two gestures. We adopted the grammar graphs for Suturing and Needle Passing from and included an additional directed link from G1 to G2 in Suturing (see Figures 1 and 2).

We acquired the gesture sequences performed for Suturing and Needle Passing from the JIGSAWS transcripts. Then we developed a method for checking if each gesture sequence follows the standard acceptable sequence of gestures in the grammar graph. As shown in Algorithm 1, for each gesture we check if it is in the grammar graph for the task and if it is a valid successor of the previous gesture, otherwise it is marked as a procedural error. Each transcript can have multiple, possibly sequential, procedural errors. This algorithm, combined with a gesture segmentation algorithm, can be used for automated detection of procedural errors in real-time.

Deviations from the grammar graph might also happen because of variations in surgical style and expertise, as discussed in Section 3.2.

3 Results

This section presents a summary of results and observations from our analysis of the JIGSAWS dataset.

8

Hutchinson et al.

3.1 Executional Errors

Table 1 lists the number of examples of each error mode as well as the total number of erroneous examples for each gesture. Note that a gesture example could exhibit multiple error modes, so the sum of the total number of errors does not necessarily equal the number of erroneous gestures.

3.1.1 Distribution of Executional Errors Among Gestures Figure 3 shows the distribution of errors of each type for each gesture from Table 1. If a gesture example had more than one error label, it was counted under the “Multiple errors” category. We

Made The Following Observations:

– G5 for both tasks and G1, G8, and G9 for Needle Passing did not have enough examples of executional errors, so further analysis was not performed on these gestures. G8 from Needle Passing and G5 from both tasks had the lowest percentage of errors because they may be less challenging than other gestures.

– G2 and G3 have the most “Multiple attempts” errors in both tasks because they require a high level of accuracy in positioning and driving the needle though the tissue, respectively. G2 has more errors in Needle Passing because the eye of the ring is a smaller target than the dot on the fabric. G3 has more errors in Suturing because surgeons often tried multiple times to align the tip of the needle with the exit point while the needle was not visible beneath the fabric.

Comparatively in Needle Passing, the needle only had to pass through one point and was always visible. Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments

Right Gripper Angle

Table 3: Kinematic Parameters with the Greatest KL Divergence Distinguishing Errors in Different

Gestures

– G4 and G6 from both tasks, and G8 from Suturing have the most gestures with multiple errors. G4 and G8 both involve manipulating the needle between the graspers and the predominant error modes were “Needle orientation” and “Multiple attempts” likely due to issues with hand coordination. For G6, the main error modes were “Out of view” and “Multiple attempts” due to multiple attempts at grasping the needle and pulling it through the ring or tissue and then moving off-camera to pull the suture through.

– G6 has a large number of “Out of view” errors especially in Suturing possibly because surgeons could not move the camera for the trials in the JIGSAWS data set. However, a different technique to pull the suture could have been used such as hand-over-hand or the pulley method that would have kept the tools within view.

3.1.2 Kinematic Parameters for Distinguishing Errors in Each Gesture We performed a comparative analysis of KL Divergence values for parameters in each gesture and identified the kinematic parameters that are associated with error occurrence as listed in Table 3.

The following are key observations from this analysis: – For G1 in Suturing, the predominant error mode was “Multiple attempts” at picking up the needle. Figures 4b and 5 show that erroneous gestures exhibited a second opening and closing of the grasper and a large difference in Y Position trajectories. This explains the large KL Divergences for those right hand parameters in Figure 4a.

– For G2 in Needle Passing, Figure 6 shows a large difference in KL Divergence for Left Rotational and Linear Velocities which may be due to the active role the left hand plays in stabilizing the ring unlike in Suturing. This is an important contextual difference between tasks.

– The main error mode for G3 was “Not moving along the curve/ Multiple attempts”. Erroneous gestures in Suturing were caused by lateral, instead of characteristically rotational, movements of the needle while in the fabric. In surgery, lateral movements may tear tissue and contribute to a safety-critical event. This explains the high KL Divergences for the parameters listed in Table 3 and shown in Figure 7a and is consistent with who found that the rate of orientation change during needle insertion (i.e. Rotational Velocity during G3) was higher for experienced surgeons.

However, Needle Passing shows nearly the opposite result in Figure 7b. Upon reviewing the gesture clips for both tasks, we noticed that clips for Suturing showed the right grasper driving the needle through the fabric and the left grasper pulling it through, but clips for Needle Passing began with the needle halfway through the ring and only showed the left grasper pulling the

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.