Surgical Instrument Tracking Rfid

Peer-Reviewed And Accepted At Cvpr 2025

CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools

3Ihu Strasbourg, France

Project page: https://github.com/camma-public/cholectrack20

Left Hand (Aslh)

Figure 1. Illustration of multi-perspective tracking in surgical domain and CholecTrack20 dataset labels for surgical tool tracking.

Abstract

Tool tracking in surgical videos is essential for advancing computer-assisted interventions, such as skill assessment, safety zone estimation, and human-machine collaboration.

However, the lack of context-rich datasets limits AI appli- cations in this field. Existing datasets rely on overly generic tracking formalizations that fail to capture surgical-specific dynamics, such as tools moving out of the camera’s view or exiting the body. This results in less clinically relevant trajectories and a lack of flexibility for real-world surgical applications.

Methods Trained On These Datasets Often

struggle with visual challenges such as smoke, reflection, and bleeding, further exposing the limitations of current approaches.

We Introduce Cholectrack20, A Specialized

dataset for multi-class, multi-tool tracking in surgical procedures. It redefines tracking formalization with three perspectives: (1) intraoperative, (2) intracorporeal, and (3) visibility, enabling adaptable and clinically meaningful tool trajectories. The dataset comprises 20 full-length surgical videos, annotated at 1 fps, yielding over 35K frames and 65K labeled tool instances.

Annotations Include Spatial

location, category, identity, operator, phase, and scene visual challenge. Benchmarking state-of-the-art methods on CholecTrack20 reveals significant performance gaps, with current approaches (< 45% HOTA) failing to meet the accuracy required for clinical translation. These findings motivate the need for advanced and intuitive tracking al- gorithms and establish CholecTrack20 as a foundation for developing robust AI-driven surgical assistance systems.

The dataset is released under CC-BY-NC-SA 4.0 license and is available for download through the project page.

1. Introduction

The true impact of computer vision research lies in its prac- tical applications, especially in critical fields like health- care. Among these, surgery represents one of the most de- manding domains, providing a definitive test for the capa- bilities of vision technologies. Surgical data science, on its own, has significantly advanced interventional health- care by leveraging data-driven techniques to provide crit- ical decision support to medical professionals . A key area of this advancement is the analysis of endoscopic video data, which offers real-time insights into surgical proce- dures, aids in skill assessment, and helps predict compli- cations [9, 56, 65]. Accurate tracking of surgical tools is central to these analyses, as it guides temporal progres- sion of procedural phases and correlates with surgical ac- tions and management of adverse events [38, 59]. Despite progress in computer vision, research shows that deep learning models pretrained on general datasets often strug- gle with surgical contexts due to complex scene dynamics, diverse tool types, and challenging visual conditions such as bleeding, smoke, and variable lighting [12, 20, 51]. This highlights the need for domain-specific datasets tailored to the unique requirements of surgical tool tracking. Obtain- ing medical and surgical data for research is challenging due to ethical and practical constraints, and annotating it re- quires expert knowledge. Current methods for tool tracking largely focus on Single Object Tracking (SOT) , Multi- Class Tracking (MCT) with one tool per class [36, 37], or Multi-Object Tracking (MOT) treating all tools as a single class [16, 46]. However, these approaches often miss the complexities of Multi-Class Multi-Object Tracking (MC- MOT) specific to surgical contexts, where tools interact dy- namically and may move out of the camera’s field of view or within the body cavity.

Multi-perspective tracking addresses these challenges by defining tool trajectories across different viewpoints dur- ing surgical procedures. It includes three critical perspec- tives (illustrated in Fig. 1): (1) intraoperative - covering the entire procedure duration to monitor tool usage and assess surgical proficiency; (2) intracorporeal - focusing on tool tracks within the body to evaluate specific tasks and predict risks; and (3) visibility - tracking tools within the camera’s field of view to provide real-time feedback to surgeons. Ex- isting surgical tracking datasets [18, 53] often lack these levels of granularity and adaptability needed for compre- hensive tool modeling. They generally follow generic track- ing formalizations and struggle to capture the intricacies of surgical tool interactions, particularly when tools are re- placed or move beyond camera visibility [6, 28, 53].

To Address This Gap, We Introduce Cholectrack20,

a novel dataset, based on laparoscopic cholecystectomy surgery, designed for multi-class multi-tool tracking from intraoperative, intracorporeal, and visibility perspectives.

Derived from raw laparoscopic videos [39, 56], it includes detailed annotations such as spatial coordinates, tool cate- gories, track identities (IDs), visual challenges, phase la- bels, and other scene attributes.

This Dataset Enhances

benchmarking resources for computer vision research and supports the development of AI models tailored to surgi- cal tool tracking, phase recognition, and surgeon perfor- mance assessment. This paper provides a comprehensive overview of the data acquisition and annotation methodol- ogy, detailed data analysis, and technical validation. In ad- dition, we conduct extensive benchmark experiments using state-of-the-art deep learning methods for object detection and tracking, evaluating performance across various surgi- cal phases and visual challenges, discussing insightful find- ings.

In Summary, The Main Contributions Are:

1. Introduction of CholecTrack20, a pioneering dataset for multi-perspective tracking with extensive annotations. 2. Extensive experimental analysis validating the dataset’s effectiveness for surgical tool detection and tracking.

3. Insights from model performance analysis under diverse visual challenges, highlighting the utility of each track- ing perspective for AI-driven surgical solutions.

2. Related Works

Object detection and tracking.

Advances In Object

detection and tracking have been driven by datasets like

, Dancetrack , And Tao , Enabling Progress

in Single Object Tracking (SOT) , Multi-Object Track- ing (MOT) [64, 66], and Multi-Class Multi-Object Tracking (MCMOT) [13, 26]. Applying these techniques to surgi- cal tool tracking poses challenges especially in the phase of bleeding, smoke, rapid movements, and variable lighting.

Traditional tracking, centered on visibility, struggles when tools leave the camera’s view or are replaced during surgery. Surgical tool tracking and configurations.

While Elec-

tromagnetic and optical tracking methods [8, 17] have been explored in surgical domain, image-based approaches better align with surgeons’ view but face issues like identity fragmentation, identity switch, and low tracking accuracy [6, 16, 28, 36, 37, 46] especially with mid-procedure tool replacements and tools exiting/re-entering the field of view or body cavity. To fully capture tool usage complexity, it is crucial to consider multiple perspectives on tool trajectories.

Existing datasets [16, 41] provide insights but lack compre- hensive coverage of these scenarios, highlighting the need for detailed datasets addressing these complexities.

3. Methodology

CholecTrack20 is a detailed dataset for surgical tool track- ing in laparoscopic cholecystectomy, offering binary and

2

spatial annotations for tools, including identity, category, bounding box location, motion, operator, phase, activity, us- age conditions, and visual challenges, essential for training and benchmarking surgical AI tool tracking algorithms.

3.1. Data Acquisition And Collection

Data source. The raw videos are sourced from the publicly

Available Cholec80 And Cholect50 Datasets, With

appropriate permissions and adherence to license terms. aid of laparoscopic cameras, these videos document laparo- scopic cholecystectomy surgeries.

Video selection.

Long Videos Were Systematically Cho-

sen to represent surgical complexities and tool variability, capturing all key phases of laparoscopic cholecystectomy. High-quality videos with clear visuals were selected, sub- sampled from 25 FPS to 1 FPS for annotation, maintaining temporal consistency and clarity.

Sensitive data handling. To protect patient and staff iden- tities, out-of-body frames potentially revealing sensitive in- formation, such as the identity of patients, clinical staff, or the operating room, were checked and anonymized follow- ing established techniques . This ensures compliance with ethical standards and privacy regulations.

3.2. Track Formalization

Given a video dataset D = {S1, S2, . . . , Sn} of laparo- scopic surgeries, each sequence Si includes frames anno- tated with bounding boxes B = [B1, B2, . . . , BM] for tool locations and classes C = {C1, C2, . . . , CN}. Tools,

=

[P1, P2, . . . , PM], are assigned unique track identities (ID) through time.

The Id Reassignment Is Guided By Visual

cues and clinical knowledge. Visually, it is based on class c ∈C and location b ∈B. Contextually, it considers the role and hand position of the surgeon operator linked to a unique trocar port p ∈P. Tool tracking solves the associ- ation matrix A(t), where Ai,j = 1 indicates the i-th tool in frame t is associated with the j-th tool in frame t + 1, and Ai,j = 0 otherwise. The aim is to obtain tool trajecto- ries T = {T1, T2, ..., TK}, each uniquely identified by ID, taking into consideration the visibility, intracorporeal, and intraoperative use.

3.3. Multi-Perspective (Mp) Trajectory

Defining tool trajectories in surgical procedures necessitates a unique approach, given the variability in tracking across

Different Perspectives, Formalized As Follows:

1. Intraoperative trajectory: This lifelong tracking starts with a tool’s first appearance and ends at its last in a pa- tient’s body during a procedure. It requires re-identification

3: Visibility Trajectory

Figure 2. Multi-perspective trajectories of surgical tool. post-occlusion, out-of-camera view, or reinsertion. This ap- proach is vital for applications such as tool usage monitor-

Ing , Inventory Management , Surgeon Training ,

skill assessment , and tool usage pattern analysis . 2. Intracorporeal trajectory: Unique tracks begin when a tool enters the body and end when it exits through a trocar port, even if off-camera. If a tool exits outside the cam- era’s view, the exit is inferred if another tool enters through the same trocar or the initial tool releases its grasp. This is essential for understanding surgical workflow, as some ac- tions occur outside the camera’s focus, like graspers holding tissue out of view to facilitate other tools’ actions . In- tracorporeal tracking supports a range of AI tasks including action evaluation , skill assessment [16, 27], tool usage optimization , and surgical risk estimation .

3. Visibility trajectory: Tracking starts with a tool’s first appearance within the camera view and ends when it leaves the view. It requires re-identification (re-ID) after occlu- sions or brief periods of disappearance within a two-second tolerance. This method is useful for assessing surgeon ac- tions and skill training [14, 24], providing go-no-go decision support, and measuring economy of motion .

Some existing studies [16, 37] follow the intraoperative tracking format, others [24, 46] employs visibility trajec- tory. The intracorporeal trajectory, being the most complex to annotate, are not well-represented in the literature. Our dataset is the first to provide fine-grained labels for all the three (Fig. 2), enabling statistical analyses such as tool us- age counts, events counts, abdominal entries/exits, tool idle- ness, out-of-camera view occurrences, and mean tracklets per perspective.

3.4. Data Annotation

Annotators and tools.

Bounding Boxes Were Annotated

by four researchers skilled in surgical workflow analysis, supplemented by prior study data [37, 40, 57]. Annota- tion tools used include Annonymized System, and a custom Python tool for visualization, merging, and validation. A pre-designed guide, refined by surgical experts, described the labels and provided image guidance as needed.

Label types and categories. Tool spatial positions were annotated using bounding box coordinates, while other la-

Hook

grasper .

Crowded

Figure 3.

Examples Of Images From Cholectrack20 Tracking

dataset with the labels overlaid on the raw images. bels were represented by class identities. Seven predomi- nant tool categories were defined: (1) cold grasper, (2) bipo- lar grasper, (3) monopolar hook, (4) monopolar scissors, (5) clipper or clip applier, (6) irrigator or suction device, and (7) specimen bag. Four operator categories were de- fined: (1) main surgeon left hand MSLH, (2) main surgeon right hand MSRH, (3) assistant surgeon right hand ASRH, and (4) null operator NULL. The assistant surgeon left hand ASLH holding the endoscopic camera is unreported. A to- tal of eight visual challenges were noted: (1) blurring, (2) bleeding, (3) camera lens fouling, (4) crowded scene, (5) occlusion, (6) smoke, (7) specular light reflection, and (8) trocar view or under-coverage. The seven commonest sur- gical phases were annotated: (1) preparation, (2) calot tri- angle dissection, (3) gallbladder dissection, (4) clipping & cutting, (5) gallbladder packaging, (6) cleaning and coagu- lation, and (7) gallbladder extraction.

Annotation process. Annotations involved drawing bound- ing boxes [x, y, w, h] over tooltips tagged with tool class c ∈C and operator class p ∈P, following trocar ports for accurate surgeon identification. Surgical details such as phase, out-of- view statuses (camera/abdomen), tool en- try/exit, and visual challenge attributes were annotated to aid accurate track assignment. Tool-tissue interaction labels from CholecT50 provides additional help in perpetuat- ing track identities. Annotations were reviewed at 25 FPS in uncertain cases, and underwent rigorous quality control, ensuring high-quality labels over two years. Fig. 3 presents samples of images from the CholecTrack20 dataset along- side their respective annotations illustrating the meticulous labeling system employed to ensures a rich dataset for de- tailed surgical tool tracking analysis.

3.5. Quality Assurance

Label agreement. Two label agreement metrics validate dataset quality: Jaccard Index for spatial overlap of bound- ing boxes and Cohen’s Kappa Statistic for category labels.

Avg. Track Length (Avg. Num. Track)

Figure 4. Dataset statistics on the distributions of (a) surgical scene visual challenges across data splits (b) track labels across perspectives, averaged across videos. Track length in seconds.

The findings of our three validation forms include: 1. Intra-rater agreement involves self-correction. We ob-

Serve A Jaccard Index Of 99.4% And Cohen’S Kappa

Scores of 94.6% for tools and 94.0% for operators. 2. Inter-rater agreement is evaluated on 20 random sam- ples across raters. The Jaccard Index is 91.8%, while tool class labels achieve 95.2% and operator labels 92.7% Cohen’s Kappa. Minor differences reflect high- quality annotations.

3. Label mediation uses a board-certified surgeon for am- biguities, particularly in operator labels. Out of 758 un- certain samples, 133 needed correction post-mediation.

3.6. Data Statistics

Quantitative overview. The dataset includes 20 surgical videos totaling over 14 hours, averaging 42 minutes per surgery. As detailed in Tab. 1, annotations cover 35,000 frames at 1 FPS (∼875, 000 at 25 FPS), with 65,000 bound- ing box labels, averaging two tools per frame. Fig. 4(a) summarizes track configurations across perspectives, show- ing tool usage by type and trajectory. For example, graspers average 27 minutes 39 seconds inside the body and are re-inserted 8.4 times, while irrigators are used for about 1 minute 30 seconds per surgery. The visibility perspec- tive contributes approximately 2,000 trajectories. Fig. 4(b) outlines tool attributes and visual challenges in the dataset, with occlusion as most prevalent occurring up to 23,000 seconds, bleeding totals around 18,700 seconds. This anal- ysis provides insights into the varied visual challenges en- countered in laparoscopic cholecystectomy.

Evaluating

tracking methods across these challenges will reveal their strengths and weaknesses. Statistical details across dataset splits are presented in Fig. 4(b).

Dataset comparison. Existing publicly available datasets primarily focus on single perspective trajectory [16, 46]. In contrast, our dataset introduces a novel approach by annota- tion different perspectives, as presented in Tab. 1, including visual challenges, phase details, activity labels, etc.

4

Table 1. Dataset comparison showing the scope, statistics, and attributes. Dataset marked ‡ are not full-length videos.

✔

✔

✔

Dataset split. The dataset is split at the procedure level into non-overlapping training, validation, and testing sets in a 5:1:4 ratio, preventing data leakage. Video distribution is balanced by procedure duration to ensure similar complex- ity and difficulty levels across all splits.

3.7. Data Analysis

We conduct a comprehensive analysis of the dataset to gain insights into label alignments and feature similarities, re- vealing correlations across labels. This would guide data preprocessing and feature selection when using the data.

Our analysis as illustrated in Fig. 5, Fig. 6, and Fig. 7 en- compasses four distinct dimensions and discussed further. Tracking vs. surgical tool type correlation. This anal- ysis explores the relationship between different tool types and their tracks, providing insights into unique patterns as- sociated with each tool during surgery. Fig. 5(a) illustrates center point locations of tools, color-coded by category over time in three videos. This shows that while some trajecto- ries appear separable, they are mostly densely interwoven, suggesting the need for advanced modeling in tool tracking.

Tracking vs. surgeon tool operator correlation. We an- alyzed the alignment between tool operator identities and tool locations, revealing dynamic interactions between sur- geons and instruments. Fig. 5(b) shows a strong correlation between operators and their tools, attributed to the distinct positioning of trocars. This underscores the value of opera- tor information in accurate track label generation.

Tracking vs. surgical phase segmentation. This analysis delves into tracking tools across surgical phases, uncover- ing how tool utilization varies with procedural stages. Fig. 6 shows that complex phases, such as calot triangle and gall- bladder dissection, exhibit densely packed trajectories due to prolonged duration and frequent tool manipulation. Sim- pler phases, like preparation and clipping, feature fewer tra- jectories, facilitating modeling. These insights inform deep learning model assessment and AI model development for surgical tool tracking.

Tracking variance. Using Exponential Moving Average (EMA), we analyze tool tracking data, as shown in Fig. 7. By overlaying EMAs on tool trajectories, we visualize vari-

(B) Tool Operator Label Features

Figure 5. 3D visualization of label alignments showing the tool position over track time. The coloring is for grouping features according to: (a) tool classes and (b) tool operators.

(F) Cleaning And Coagulation

Figure 6. 3D visualization of tracking across different surgical phases for some randomly selected videos.

Ema

Figure 7. Trajectories of selected tools with EMA over time. Plot- ted positions are computed as weighted combinations of the center coordinate of the bounding boxes, scaled by image size.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.