Simulia Haptic Surgical Training

A Surgical Foundation Model Reveals Task-Dependent Label Efficiency Florian Philipp Stilza,b,c,∗, Lorenzo Arboita,b, Vinkle Srivastava,b,d, CAMMA International Surgical Partners, Jacques Marescauxe, Sergio Alfierif, Pietro Mascagnib,f, Nassir Navabc, Nicolas Padoya,b

Bihu Strasbourg, Strasbourg, France

eIRCAD, Research Institute Against Digestive Cancer, Strasbourg, France fFondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Italy Developing label-efficient models is a central challenge in surgical AI due to the high cost and scarcity of expert annotation.

While self-supervised foundation models adapt well to new tasks with minimal data, how label efficiency varies across different surgical tasks remains largely unexplored. Here, we introduce SURGE, a surgical foundation model trained on SurgSpectrum-30M+, the largest pretraining dataset comprising over 30 million frames, with checkpoints released to enable further research. We systematically evaluate label efficiency across 5 task categories and 15 benchmarks. These range from temporal and spatial scene understanding to fine- grained reasoning tied to instrument–anatomy interactions and safety-critical maneuvers.

SURGE outperforms prior state-of-the-art on all benchmarks, even surpassing task-specific models on complex reasoning tasks. Crucially, we reveal a task-dependent scaling behavior: while scene understanding tasks saturate with minimal supervision, fine-grained reasoning tasks continue improving with substantially larger annotation budgets, providing a blueprint for allocating expert effort in complex domains.

Code: https://github.com/CAMMA-public/SURGE.

1. Introduction

Artificial intelligence is increasingly deployed to support expert decision-making in complex, high-stakes domains, ranging from healthcare and science to engineering and pub- lic safety . In these settings, models are typically trained on annotated datasets that require substantial input from do- main experts. However, generating such annotations is both time-consuming and costly, as expertise is scarce and tasks often demand deep contextual understanding. As a result, the development of new AI systems is frequently constrained not only by model capacity, but increasingly by the availability of high-quality labeled data . Improving the efficiency with which new tasks can be learned is therefore critical. This can be approached from two complementary directions: develop- ing models that achieve strong performance with fewer anno- tations and understanding how annotation requirements vary across tasks to allocate expert effort more effectively. Here, we address both challenges by developing a self-supervised foun- dation model for surgery and systematically characterizing how its performance scales with the amount of task-specific labeled data. We use this scaling behavior to quantify label efficiency across surgical perception and reasoning tasks, pro- viding a principled basis for understanding and reducing the annotation requirements of AI systems in surgery.

Surgical procedures are complex, high-risk interventions that require continuous perception and timely reasoning to support safe and effective decision-making. Automated anal- ysis of endoscopic video acquired during minimally inva- sive surgery has the potential to improve intraoperative guid- ance, surgical training and postoperative assessment.

Sur-

gical AI [2, 3, 4, 5] is therefore a particularly high-stakes application, as errors and preventable complications during surgery contribute substantially to global morbidity and mor- tality [6, 7], highlighting the need for reliable intraoperative decision support. Importantly, surgical video analysis encom- passes multiple levels of understanding, ranging from tem- poral perceptual tasks, such as recognizing surgical phases and steps [8, 9, 10, 11, 12] and spatial tasks such as seg- menting instruments and anatomy [13, 14, 9, 11, 15, 16, 17], to higher-level reasoning about instrument–anatomy interac- tions [18, 19] and clinically grounded decision-making .

These diverse tasks vary substantially in complexity, raising the question of whether they also differ in the amount of su- pervision required to learn them effectively.

Recent advances in self-supervised and foundation model pretraining [20, 21, 22, 23] have enabled strong general- purpose visual representations by learning from large amounts of unlabeled data.

When Applied To Surgical Video, These

approaches often yield competitive performance, particularly when sufficient labeled data is available [24, 25, 26]. Domain- adapted pretraining [24, 25, 27, 26, 28, 29] and multimodal approaches [30, 31] have further improved data efficiency.

However, existing work has primarily focused on developing stronger representations, while the extent to which different categories of surgical tasks exhibit distinct label efficiency re-

B

Fig. 1: a: Label efficiency across surgical tasks, shown as normalized area under the curve (nAUC) in low label regimes versus relative peak performance. Results are averaged across datasets per task. We observe a consistent ordering of tasks by data efficiency: perception tasks (segmentation (Seg), Phase, Steps) achieve strong performance in low-label regimes, while fine-grained and reasoning tasks (Action Triplet, CVS) require substantially more supervision. Notably, certain tasks exhibit low-data regimes with limited learning signal, indicating threshold effects that are not overcome by pretraining. b: Full-data performance comparison of our models with general-domain and surgical foundation baselines across 15 benchmarks. c: Five tasks are illustrated with representative examples and grouped into perception and clinical reasoning categories. Icons indicating anatomical structures denote the surgical procedures associated with each task (cholecystectomy, hysterectomy, liver resection, gastric bypass, prostatectomy). The nAUC metric is illustrated schematically.

3

mains largely unexplored. In this work, we address this gap through a large-scale, systematic analysis of label efficiency in surgical AI. To this end, we introduce SURGE (Surgical Understanding through Representation and Generalization at Scale), a sur- gical foundation model trained via self-supervised learning on SurgSpectrum-30M+, a dataset of over 30 million sur- gical video frames.

Using This Unified Representation, We

evaluate performance across a spectrum of tasks, which we group into perceptual tasks (e.g., spatial and temporal under- standing) and clinically grounded reasoning tasks (e.g., re- lational instrument–anatomy interactions and safety-critical maneuver recognition), which require higher-level judgment built upon perceptual understanding. Beyond overall perfor- mance, we study how these tasks scale with increasing su- pervision. While large-scale domain-adapted pretraining sub- stantially reduces labeled training data requirements, it does not homogenize performance. Instead, tasks exhibit distinct label efficiencies that persist even under stronger supervi- sion and higher-parameter adaptation, suggesting that the effi- ciency with which different aspects of surgical understanding can be learned depends not only on representation quality, but also on the structure of the task itself.

2. Results

Understanding label efficiency in surgical computer vision requires disentangling the interplay between representation quality, task structure, and supervision.

To this end we build on large-scale domain-adapted self- supervised pretraining, training ViT-Base and ViT-Huge+ on SurgSpectrum-30M+ (over 30 million surgical video frames), resulting in SURGE-B and SURGE-H, providing a strong and unified visual representation for surgical video analysis. As a result, SURGE provides a well-suited foundation for studying label efficiency, as it reduces performance bottlenecks associ- ated with weaker or less robust representations. This unified representation enables us to study supervision scaling while reducing confounding effects from representation quality.

We evaluate these representations across a diverse set of surgical tasks, spanning two main categories: surgical percep- tion and more fine-grained and clinically grounded reasoning tasks. Surgical perception tasks are separated into temporal workflow understanding at different granularities (phase and step recognition) and spatial perception (semantic segmenta- tion), while reasoning tasks focus on analyzing surgical exe- cution. These reasoning tasks include clinically grounded as- sessments (Critical View of Safety recognition) as well as relational instrument-anatomy understanding (action triplet recognition), which has shown correlations with patient out- comes . Action triplet recognition refers to predicting sur- gical actions as ⟨instrument, verb, target⟩tuples of tool-tissue interactions.

We systematically vary the amount of supervision, from a single training video to the full training set for each task, and evaluate all models using a unified linear-probing protocol in which only a linear classifier is trained on frozen representa- tions. This protocol isolates the contribution of representation quality to downstream label efficiency. We compare SURGE with general-domain foundation models and prior surgi- cal self-supervised approaches [24, 28, 26]. Across tasks, per- formance improves consistently with increasing supervision, while stronger representations provide substantial gains, par- ticularly in low-label regimes.

We quantify label efficiency using normalized area under the curve (nAUC; see Methods and see Fig. 1c). We compute nAUC in two settings: over the full training budget and in a low-label regime consisting of 1 and 3 training videos.

2.1. SURGE outperforms other foundation models across sur-

Gical Ai Tasks

SURGE achieves the best performance across all 15 bench- marks spanning diverse surgical tasks and supervision regimes (Fig. 1, Figs. 3, 4). For the subsequent label-efficiency analy- sis, we consider 14 benchmarks for which supervision can be systematically varied at the video level. AutoLaparo-Seg is evaluated using its official train/validation/test splits only, as its data cannot be partitioned into individual videos for con- trolled supervision scaling. We illustrate this overall advan- tage with representative results from temporal workflow un- derstanding and fine-grained relational reasoning.

For temporal workflow understanding, SURGE-B surpasses the strongest baseline on every dataset, achieving an aver- age F1 score of 69.40 (95% CI: 63.30-75.85), compared with 63.63 (95% CI: 57.33-70.08) for the strongest competing foundation model. Scaling the representation from SURGE-B to SURGE-H further improves performance to 72.23 (95% CI: 66.18-78.51), demonstrating consistent gains with increased model capacity.

The advantage extends beyond temporal perception to more fine-grained relational reasoning. On instrument-anatomy in- teraction recognition, SURGE-B achieves 38.72 mAP (95% CI: 34.3-43.13), compared with 32.87 (95% CI: 28.6-37.13) for the strongest per-dataset baselines.

Surge-H Further

increases performance to 41.40 mAP (95% CI: 38.6-44.2). Thus, domain-adapted pretraining consistently improves sur- gical video representations across both temporal perception and fine-grained relational understanding, with further gains from scaling model capacity.

2.2. Consistently superior label efficiency across all annota-

We Next Examine Whether The Performance Gains Of

SURGE translate into greater label efficiency across the full range of annotation budgets (Fig. 2). Averaged across datasets, SURGE-H achieves an nAUC of 0.914 (95% CI: 0.884–0.942), substantially outperforming the strongest base- lines on each dataset with 0.769, (95% CI: 0.724–0.809).

This advantage persists in low-label settings (Fig. 2), where SURGE-H attains a low-label nAUC of 0.757 (95% CI: 0.683–0.835) versus 0.624 (95% CI: 0.549–0.692).

We further analyze the effect of model scale. SURGE-B and SURGE-H contain 85.67M and 840.63M parameters, respec- tively, and larger models consistently improve performance, with SURGE-H outperforming SURGE-B in 14 out of 15

Low-Label & Overall Label Efficiency

Fig. 2: Overall and low-data regime label efficiency comparison: measuring normalized AUC (nAUC) across tasks/datasets, highlighting most efficient model both in heatmap color coding and box highlighting. benchmarks, as depicted in Fig. 1b. This improvement is es- pecially pronounced in low-label settings; while task variance results in wide confidence intervals, paired analysis reveals a highly significant and substantial performance gain for the larger model (SURGE-H: 0.757 (95% CI: 0.683–0.835) and SURGE-B: 0.696 (95% CI: 0.619-0.769)); Wilcoxon signed- rank p < 0.01, Cohen’s d = 1.25).

Overall, SURGE substantially improves label efficiency in surgical AI, enabling strong performance with reduced anno- tation effort. These gains are consistent across five surgical procedures (Fig. 2), although the magnitude of improvement varies across tasks, reflecting differences in how performance scales with supervision.

2.3. Domain adaptation bridges general and surgical repre-

Sentation Strengths

We next examine why SURGE performs consistently across diverse surgical tasks.

Prior Surgical Self-Supervised Ap-

proaches are particularly effective on workflow and clinically grounded tasks (Critical View of Safety), especially in low- data regimes, but are less effective for dense spatial percep- tion and fine-grained instrument-anatomy interaction seman- tics. In contrast, general-domain self-supervised models pro- vide stronger representations for tasks such as semantic seg- mentation and instrument-anatomy interaction (Fig. 2).

SURGE combines these complementary strengths. It re- tains the strong performance of surgical self-supervised ap- proaches on workflow and clinically grounded tasks while exceeding general-domain models on dense spatial and interaction-level tasks.

For Example, Surge-H Achieves

Fig. 3: Coarse temporal perception (phase recognition) performance curves: All subset runs are averaged over 3 runs with error bands highlighting the standard deviation.

D

Fig. 4: Additional perception & reasoning task performance curves: a shows spatial understanding (semantic segmentation), b: fine temporal un- derstanding (step recognition), c: safety maneuver analysis (CVS), and in d instrument-anatomy interaction (action triplet). All subset runs are averaged over 3 runs with error bands highlighting the standard deviation.

82.95 mIoU on GraSP, compared with 76.09 for DINOv3- H, and 44.20 mAP on CholecT50, compared with 37.13 for DINOv3-H. These results suggest that large-scale domain adaptation enables representations that capture both general visual structure and fine-grained surgical semantics, provid- ing a potential basis for SURGE’s broad transferability and label efficiency.

2.4. Perception tasks exhibit high label efficiency with task

Structure Dependent Variations

Our analysis reveals that surgical perception tasks exhibit high label efficiency, with performance saturating under min- imal supervision.

Tasks That Rely Primarily On Visual Cues

or coarse temporal structure achieve strong performance with very limited annotated data, with diminishing gains as more data is added.

For Instance, In Semantic Segmentation On

prostatectomy (GraSP dataset), SURGE-H attains an mIoU of 76.68 ± 2.53% using only a single training video, with perfor- mance remaining nearly unchanged as the training set scales to eight videos as shown in Fig. 4a. Similarly, in phase recog- nition, SURGE-H reaches 74.04 ± 2.09% F1 on gastric bypass (StrasBypass70) with one video and quickly plateaus with ad- ditional data, while achieving comparable behavior on chole- cystectomy (Cholec80) depicted in Fig. 3. In these settings, a single labeled video often suffices to outperform compet- ing pretrained surgical models trained on substantially larger datasets.

While this regime of high label efficiency is consistent across perception tasks, we find that structural properties of the task modulate the rate at which saturation is reached. In- creasing the granularity of the prediction objective introduces a more pronounced dependence on labeled data. On Stras- Bypass70, while coarse phase recognition improves modestly with additional data (8.5% F1 gain from 1 to 40 videos), the more fine-grained step recognition task exhibits a substantially steeper scaling curve, improving by 17.3% F1 over the same range seen in Fig. 4b. Thus, increasing temporal granularity substantially increases the amount of supervision required to reach comparable performance.

Similarly, the complexity of the annotations influences data requirements even within this otherwise efficient regime. Comparing semantic segmentation across two datasets (GraSP and Endoscapes-Seg50) in Fig. 4a illustrates this effect.

GraSP reaches strong performance quickly with limited data, whereas Endoscapes-Seg50 improves more gradually, starting from a lower baseline and continuing to benefit from addi- tional annotations. This difference reflects the nature of the labels: GraSP focuses on a smaller set of well-defined struc- tures, primarily surgical instruments, whereas Endoscapes- Seg50 includes a larger and more fine-grained label space, including small anatomical structures alongside instruments, resulting in a more complex prediction problem. As a result, the model must learn a more complex mapping from visual features to labels, which increases the amount of data required to achieve comparable performance.

Taken together, these results show that surgical perception tasks form a regime of high label efficiency characterized by rapid saturation with minimal supervision. Although fac- tors such as task granularity, annotation density, and semantic complexity modulate the precise label efficiency, they do not alter the overall trend of efficient learning in these tasks. In the next section, we show that this regime breaks down for reason- ing tasks, which exhibit distinctly different label efficiency and remain far from saturation even under substantially increased supervision.

2.5. Reasoning tasks exhibit persistent data requirements In contrast to the rapid saturation observed in perception tasks, reasoning tasks exhibit fundamentally different label ef- ficiency, remaining far from saturation even under substan- tially increased supervision.

In the instrument-anatomy interaction tasks (action triplet recognition) on the CholecT50 and ProstaTD benchmarks shown in Fig. 4d, SURGE-H maintains a consistent lead over all baselines, yet displays a remarkably persistent log-linear scaled curve that still allows for sustained performance gains under constant increase in labeled data.

For Instance, On

ProstaTD, SURGE-H begins at 20.30 ± 2.53 mAP and re- quires 14 training videos to reach 38.36 mAP, a trajectory that lacks the ’jump-start’ characteristic, where 1-video per- formance is already a significant fraction of the ceiling, ob- served in temporal (phase) or spatial (semantic segmentation) perception tasks. This is also supported by the nAUC met- ric, where perception tasks achieve average scores of 0.925,

D

Fig. 5: Label instances and label efficiency comparison across tasks & higher parameter adaptation results: a shows task comparisons for prostatectomy and b for cholecystectomy. Highlighting severe annotation density differences across tasks. c&d: We select the best model for this analysis (SURGE-H) and compare linear probing (SURGE-H LP) and partial tuning of the backbone (SURGE-H PT). All subset runs are averaged over 3 runs with error bands highlighting the standard deviation. SOTA on Endoscapes is LG-CVS and on CholecT50 is TripletGCN .

0.940, and 0.919 for semantic segmentation, phase recogni- tion, and step recognition, respectively, compared to 0.832 for action triplet recognition. This difference is also present in the low-label regime, where semantic segmentation, phase recognition, and step recognition achieve average low-label nAUC values of 0.825, 0.828, and 0.750, respectively, versus 0.607 for instrument-anatomy interaction (action triplet recog- nition).

This structural bottleneck is most pronounced in the clinical quality assessment (Critical View of Safety (CVS)), depicted in Fig. 4c, on the Endoscapes dataset.

Despite Leveraging

the same large-scale pretrained representation, performance for SURGE-H starts at 27.92 ± 1.17 mAP and requires sub- stantially more labeled data, reaching 53.26 mAP only when scaling to 120 training videos. In contrast to other tasks, we observe an apparent threshold-like behavior, where all models exhibit limited learning signal when trained with 10 or fewer videos, and only begin to improve meaningfully beyond this regime. This pattern is further reflected in the label-efficiency metric (nAUC), which reaches only 0.882 when evaluated across all training budgets and drops to 0.520 in the low-label regime. Notably, the low-label nAUC is substantially lower than that observed for any other task, indicating that this task benefits less from limited supervision. This suggests that cer- tain reasoning tasks require a minimum amount of supervision before meaningful generalization can occur.

2.6. Label instances as an indicator of supervision require-

Ments

To further ground these differences in data requirements, we next quantify the underlying supervision effort associated with each task, providing a practical proxy for label efficiency.

While true annotation cost is a complex composite of me- chanical time (e.g., pixel-wise segmentation) and cognitive load (e.g., clinical consensus for safety criteria), direct mea- surement of human hours across retrospective, multi-center datasets is very difficult and annotator specific. To overcome this limitation and provide a standardized baseline, we intro- duce label instances, the absolute count of all explicitly anno- tated instances across a dataset, as an objective, macro-level proxy for total annotation effort. By systematically quanti- fying the volume of human-generated labels required per task category, we offer a first-of-its-kind approximation of compar- ative supervision demands. As shown in Fig. 5a-b, this result- ing distribution closely mirrors the previously observed scal- ing patterns, demonstrating that total label instances serves as a powerful indicator of the intrinsic human effort required to solve different surgical tasks.

Notably, more clinically relevant tasks requiring clinical reasoning and relational instrument-anatomy understanding (e.g., CVS and action triplet recognition) exhibit substantially higher label counts, reflecting significantly greater annotation effort. Despite this increased supervision, their performance continues to improve with additional annotations without clear signs of saturation. This indicates that these tasks remain data- hungry even under substantially higher annotation volumes.

In contrast, temporal and spatial perception tasks rely on comparatively fewer annotations and exhibit more favorable learning efficiency.

Among These, Semantic Segmentation

stands out with the highest number of labels per video due to its dense annotation structure, yet still benefits from com- paratively efficient learning dynamics.

Overall, these findings reveal a pronounced imbalance in supervision demands across tasks and indicate that label effi- ciency is strongly task-dependent.

7

2.7. Higher parameter adaptation surpasses supervised

Sota Without Altering Label Efficiency

Given the strong task-dependent differences in label effi- ciency observed above, we next examine whether these dy- namics can be altered through higher parameter adaptation, in which more parameters of the pretrained backbone are up- dated using task-specific supervision rather than training only a linear classifier. While higher parameter adaptation substan- tially improves absolute performance, enabling data-intensive tasks such as relational instrument-anatomy recognition and clinical reasoning to surpass task-specific supervised state-of- the-art baselines, the fundamental label efficiency with respect to supervision remains unchanged.

As shown in Fig. 5c-d, partial adaptation consistently shifts performance upwards, yet preserves the relative gains obtained from additional supervision.

Notably, This Up-

ward shift yields absolute state-of-the-art gains of 6.4% mAP on Endoscapes and 15.5% mAP on CholecT50 action triplet recognition. These improvements are achieved using our generalized approach, without relying on the complex, task-specific modifications used by previous baselines, such as segmentation-based supervision or ensemble strate- gies .

3. Discussion

Our results provide a systematic view of how surgical AI learns from expert supervision across tasks with fundamen- tally different demands.

Large-Scale Domain-Adapted Self-

supervised pretraining produces a broadly transferable repre- sentation that consistently outperforms both general-domain and prior surgical foundation models across 15 benchmarks, while providing substantial gains in low-label regimes. How- ever, our results also reveal that representation quality alone does not determine label efficiency. Surgical perception tasks typically reach high performance with minimal supervision and rapidly saturate, whereas clinically grounded and rela- tional reasoning tasks continue to benefit from substantially more supervision. Increasing the number of adapted param- eters raises absolute performance and enables state-of-the-art results on these more demanding tasks, but does not funda- mentally alter their scaling with supervision. Together, these findings suggest that representation quality determines how effectively models can exploit available supervision, whereas the structure of the downstream task determines how much supervision is ultimately required.

3.1. The Saturation Of Perception Tasks

The rapid saturation observed in spatial understanding tasks (semantic segmentation) and temporal understanding (phase recognition) suggests that the field may be approaching a mile- stone in perceptual learning. Our results show that with suf- ficiently large domain-adapted pretraining, the model’s inter- nal representation of surgical instruments, tissue textures, and broad procedural states becomes “task-ready" with very lim- ited label supervision. In these regimes, a single training video acts not as a source of feature learning, but as a “Rosetta Stone" that simply maps existing high-quality representations to class names.

This finding suggests an immediate pivot in how we al- locate annotation resources.

For Standard Procedures And

instrument-centric tasks, the pursuit of larger datasets may ex- hibit diminishing returns. Instead, these “solved" perceptual components should be leveraged as automated foundations to assist in the more difficult task of mapping complex medi- cal/surgical logic.

3.2. The persistent reasoning gap and the need for hybrid

Modeling

In contrast to the saturation of perception, fine-grained and reasoning tasks such as action triplet recognition and Criti- cal View of Safety (CVS) assessment exhibit a “Reasoning Gap” that is not fully addressed by scale alone. These tasks require the model to move beyond identification toward com- positional logic and clinical judgment. The persistent growth of the scaling curves observed in these domains indicates that the model is actively learning the underlying rules of surgery frame-by-frame, rather than solely relying on pretrained visual archetypes.

This resistance to pretraining gains does not suggest a lim- itation of the underlying representations, adaptation strategy, or model capacity, but rather may reflect the current frontier of deep learning. To cross this Reasoning Gap, we appear to require substantially more annotated data efforts compared to prior perception tasks or a transition toward hybrid method- ologies.

Such Approaches Might Include A Combination Of

largely-pretrained foundation models with task-specific archi- tectural modeling, the integration of multi-modal clinical in- puts, or the development of pretraining objectives that explic- itly target causal and relational structures to learn those com- plicated problems under limited supervision efficiently.

3.3. A Strategic Roadmap To Jumpstart Surgical Ai

By characterizing how data requirements vary across surgi- cal tasks, our work provides a practical framework for under- standing their label efficiency and guiding annotation strate- gies. This perspective enables a more principled allocation of labeling effort toward clinically impactful objectives: • Low-shot transfer for perception: Tasks such as in- strument segmentation and coarse video understanding (phase recognition) exhibit high label efficiency and ben- efit strongly from domain-adapted pretraining. For these tasks, annotation requirements can be substantially re- duced, opening the door to few-shot or weakly super- vised approaches.

• Targeted annotation for reasoning: In contrast, tasks involving relational instrument-anatomy understanding and clinical assessment remain significantly more data- intensive.

Expert Annotation Effort, Being Scarce And

costly, should therefore be prioritized toward these set- tings.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.