Stm32 Medical Device Mcu

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration Jiahui Penga, He Yaoa, Jingwen Lib, Yanzhou Suc, Sibo Jud, Yujie Lua, Jin Yee, Hongchun Luf, Xue Lib, Lincheng Jianga, Min Zhua,∗and Junlong Chenga,e,∗

A B S T R A C T

Contrastive Language-Image Pre-training (CLIP) has demonstrated outstanding performance in global image understanding and zero-shot transfer through large-scale text-image alignment. However, the core of medical image analysis often lies in the fine-grained understanding of specific anatomical structures or lesion regions. Therefore, precisely comprehending region- of-interest (RoI) information provided by medical professionals or perception models becomes crucial. To address this need, we propose MedP-CLIP, a region-aware medical vision-language model (VLM). MedP-CLIP innovatively integrates medical prior knowledge and designs a feature-level region prompt integration mechanism, enabling it to flexibly respond to various prompt forms (e.g., points, bounding boxes, masks) while maintaining global contextual aware- ness when focusing on local regions. We pre-train the model on a meticulously constructed large-scale dataset (containing over 6.4 million medical images and 97.3 million region-level annotations), equipping it with cross-disease and cross-modality fine-grained spatial semantic understanding capabilities. Experiments demonstrate that MedP-CLIP significantly outperforms baseline methods in various medical tasks, including zero-shot recognition, interactive segmen- tation, and empowering multimodal large language models. This model provides a scalable, plug- and-play visual backbone for medical AI, combining holistic image understanding with precise regional analysis.

1. Introduction

CLIP (Radford et al., 2021) has achieved remarkable zero-shot generalization across diverse vision-language applications, including open-world recognition (Gu et al., 2021; Xu et al., 2023), multimodal large language models (MLLMs) (Li et al., 2023c; Liu et al., 2023), and generative tasks in both 2D and 3D domains (Li et al., 2023b; Nichol et al., 2022; Lee and Chang, 2022). This success stems from large-scale pretraining that captures holistic image-text alignment, enabling robust understanding of global semantics. However, CLIP and its derivatives struggle in domains requiring precise, fine-grained region-specific interpretation—most critically in medical imaging, where diagnostic accuracy hinges on localized anatomical structures or pathological lesions.

Medical image analysis fundamentally differs from natural image understanding. Clinical workflows are inherently localized: diagnoses often depend on identifying subtle, spatially confined anatomical structures or pathological regions, such as tumors, lesions, or organ boundaries. Existing CLIP adaptations for the medical domain, such as BiomedCLIP (Zhang et al., 2025a), PMC-CLIP (Lin et al., 2023), and MedCLIP (Wang et al., 2022), enhance global semantic alignment by pretraining on domain-specific image-text corpora (Lin et al., 2023; Wang et al., 2022). Yet, these models still lack native region-aware capabilities and remain global encoders, limiting their utility in clinical scenarios that require precise spatial reasoning.

In pursuit of region-aware understanding, several approaches have emerged in general computer vision. Naive strategies include isolating RoIs through cropping (Subramanian et al., 2022; Zhong et al., 2022) or suppressing background context via masking (applied at either the input or feature/attention level) (Zhou et al., 2022; Liang et al., 2023). However, these methods suffer critical drawbacks: cropping inherently disrupts crucial global context, while

Arxiv:2604.11197V3 [Cs.Cv] 29 Jun 2026

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration Figure 1: MedP-CLIP can be flexibly deployed for various downstream tasks, significantly enhancing the region-aware analysis capability of medical imaging. This model supports direct application and plug-and-play integration, and accepts explicit prompts from users as well as outputs from perceptual models.

masking neglects potentially informative surrounding structures and both undermine the holistic reasoning essential for accurate medical interpretation. Alternative frameworks like Alpha-CLIP (Sun et al., 2024) attempt to mitigate this by fusing RoI information via an auxiliary alpha channel input, preserving the full image context. Nevertheless, this architectural choice inherently ties the model’s inference performance to the precision of the mask provided, exhibiting limited practicality in rapid clinical workflows due to cumbersome input requirements. Within the medical domain, methods such as R-LLaVA (Chen et al., 2025) and MedPLIB (Huang et al., 2025) leverage bounding boxes or masks to guide large language models (LLMs) in contextualizing local regions. However, these solutions typically entail intricate, task-specific multimodal pipelines that incur substantial training costs and lack scalability across diverse clinical applications.

To overcome these limitations, we introduce MedP-CLIP, a region-aware VLM designed specifically for medical image analysis. Unlike approaches that compromise global context or impose impractical input requirements at infer- ence, MedP-CLIP features a novel feature-level region prompt integration mechanism to achieve native promptability.

While we utilize high-precision pixel-level masks during pre-training to establish accurate anatomical ground truth, our architecture is explicitly designed to decouple from them during deployment. By processing sparse embeddings through cross-attention, this design natively handles low-effort, arbitrary spatial prompts (including points, bounding boxes, and masks) while preserving comprehensive image context. Crucially, we synthesize 97.3 million regions by systematically integrating mask annotations from the IMed-361M (Cheng et al., 2025) dataset. This medical-specific optimization enables flexible, precise region understanding without task-specific pipelines or complex multimodal fusion, establishing a scalable backbone for diverse clinical applications. We comprehensively evaluate the four core medical AI capabilities of MedP-CLIP (Fig. 1), demonstrating its effectiveness in region-aware medical image understanding: (1) Image Recognition (Fig. 1(a)): While preserving CLIP’s global recognition capabilities, MedP- CLIP uniquely enables prompting for arbitrary regions, improving average accuracy by 4.45% over UniMed-CLIP on unseen datasets. (2) Interactive Segmentation (Fig. 1(b)): As a plug-and-play backbone for interactive frameworks (e.g., SAM), MedP-CLIP achieves state-of-the-art segmentation performance with minimal fine-tuning costs (∼1% of total parameters). (3) Region-Level Visual Grounding (Fig. 1(c)): By extending CLIP’s image-text alignment to regional semantics, our model establishes correlations between local lesions and diagnostic narratives. (4) MLLM Clinical Reasoning (Fig. 1(d)): Integrated with LLMs, our visual encoder mitigates anatomic misidentification (e.g., kidney vs. liver structures) and enables medical vision-language QA.

Page 2 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration In summary, MedP-CLIP improves CLIP’s fine-grained spatial understanding through a feature-level region prompt integration mechanism. It seamlessly processes various prompts while preserving critical global context. Pre- trained on a meticulously curated large-scale dataset comprising 97.3 million medical region-text pairs, it achieves robust cross-modal fine-grained semantic reasoning capabilities. MedP-CLIP outperforms benchmark models across various medical tasks, including but not limited to zero-shot recognition, interactive segmentation, and empowering multimodal LLMs, thereby establishing an expressive and scalable foundation for biomedicine.

2.1. Medical Vlms

CLIP (Radford et al., 2021) has established powerful transferable representations through large-scale image-text alignment, but its application in the medical domain remains challenging due to the lack of prior anatomical knowledge and limitations in fine-grained semantic representation. To address this, domain-specific adaptation methods have been pre-trained on medical corpora. For instance, MedCLIP (Wang et al., 2022) and PMC-CLIP (Lin et al., 2023) leverage medical image-text data for specialized pre-training, enabling CLIP to align global information in medical images with relevant text. Further, BiomedCLIP (Zhang et al., 2025a) underwent large-scale training on the PMC- 15M dataset containing 15 million biomedical image-text pairs, even outperforming state-of-the-art radiology-specific models on tasks like RSNA pneumonia detection. Recently, Khattak et al. developed UniMed-CLIP (Khattak et al., 2024) on datasets spanning six medical modalities, enhancing CLIP’s multimodal generalization ability in medicine.

More recent efforts have further expanded this direction: BIOMEDICA (Lozano et al., 2025) constructs a large-scale open biomedical image-caption archive from scientific literature, MedicalNarratives (Ikezogwo et al., 2025) introduces localized medical narratives by aligning medical visual content with explanatory language and spatial traces, and SurgLaVi (Perez et al., 2026) builds a large-scale hierarchical surgical vision-language dataset with semantically enriched clip-caption pairs for surgical representation learning. Meanwhile, CLIP-Guided Generative Network (Zhang et al., 2025b) explores CLIP-guided pathology nuclei image augmentation to improve nuclei segmentation and classification under limited annotations. Nevertheless, existing medical VLMs still largely lack a native and flexible region-prompt interface for clinician-in-the-loop interaction, which may be insufficient for diagnostic workflows requiring precise localization of anatomical structures and pathological regions. Consequently, they are unable to support clinician-in-the-loop interaction requiring dynamic region-aware prompting, highlighting an unresolved gap in medical VLMs.

2.2. Region-Aware Methods In Vlms

Research on region-aware visual language models aims to address the limitations of global image representations. Early input-level adaptive methods, such as ReCLIP (Subramanian et al., 2022) and CircleCLIP (Shtedritski et al., 2023), guided model attention by cropping or overlaying regions of interest. However, such disruptive operations inevitably sacrifice critical contextual information, which is particularly detrimental in medical imaging where patho- logical structures exhibit complex spatial dependencies. Subsequent feature-level methods, including MaskCLIP (Zhou et al., 2022) and MaskAdaptedCLIP (Liang et al., 2023), preserved structures through attention masking but severely lost edge information relevant to diagnosis. While RegionCLIP (Zhong et al., 2022) improved box-level alignment and Alpha-CLIP (Sun et al., 2024) introduced auxiliary alpha channels for soft modulation, they respectively suffered from coarse spatial reasoning and an architectural reliance on clinically impractical high-precision mask annotations during inference. Recent biomedical CLIP extensions have also improved concept-level interpretability and cross- domain generalization, as CDCLIP (Li et al., 2026) decomposes diagnostic categories into human-interpretable concepts for zero-shot medical image classification, while MMKD-CLIP (Wang et al., 2025b) distills knowledge from multiple biomedical CLIP teachers to build a more generalist foundation model across modalities. In medical imaging, uncertainty-aware diagnostic phrase identification and grounding (Zou et al., 2025) has further explored the alignment between clinically meaningful diagnostic expressions and corresponding image regions, emphasizing the importance of reliable region-level medical image-text correspondence for diagnostic interpretation. Expansion in the medical domain has demonstrated the utility of spatial priors: R-LLaVA (Chen et al., 2025) enhanced region awareness by directly injecting bounding boxes into image space for visual question answering, while MedPLIB (Huang et al., 2025) supported flexible region prompts through multi-stage mixture-of-experts training. However, these methods either extended the global CLIP backbone in a task-specific manner (e.g., R-LLaVA) or required complex multi-stage training and inference pipelines (e.g., MedPLIB), thereby imposing significant limitations on scalability and practical

Page 3 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration deployment. In contrast, our MedP-CLIP addresses this issue by treating region prompts as a flexible region-prompt interface for interaction. It supports multiple spatial input modes without relying on external modules or post-hoc inference mechanisms, enabling precise and direct region-level representation within a unified architecture.

2.3. Non-Native Region Integration in Medical VLMs The imperative for spatial precision in medical diagnosis has spurred the development of models capable of directly interpreting and responding to region-specific prompts. Such models aim to fuse textual instructions with localized image semantics to simulate clinician interaction paradigms. For instance, MedRegA (Wang et al., 2025a) leverages sample-specific lesion or anatomical structure coordinates to focus attention, while

(Gao Et Al., 2025)

integrate abnormal region awareness within multimodal fusion for generating more detailed descriptions. Although these methods surpass global models in localization capability, they frequently depend on multi-stage alignment processes, constraining model flexibility. Concurrently, techniques like M2CRL (Hu et al., 2024) employ intricate masking schemes for endoscopic video analysis, and VertFound (Wu et al., 2024) combines CLIP semantics with spatial features from SAM (Kirillov et al., 2023) for fine-grained vertebra classification; however, these approaches are often confined to specific tasks or modalities. Beyond region-aware vision-language modeling, recent studies have explored uncertainty-aware segmentation (Zou et al., 2025) and uncertainty-guided multi-prompt adaptation (Zhou et al., 2025) to improve the reliability of prompt-based medical segmentation systems, further highlighting the importance of robust prompt-region interaction in medical image analysis. Furthermore, MedDAM (Xiao et al., 2025) adapts local description models to prioritize clinically critical lesion features. In parallel, FedMedCLIP (Wu and Chaddad, 2026) adapts CLIP to privacy-preserving heterogeneous medical classification by freezing CLIP encoders and communicating lightweight masked feature adapters, but its objective remains image-level classification rather than native region- prompt interaction. Critically, a common limitation persists: these solutions typically incur increased model complexity and computational overhead. More fundamentally, they rely on non-intuitive prompts (e.g., predefined coordinate systems) or auxiliary modules for spatial encoding. This dependence on specialized, non-native interfaces can limit seamless, clinician-centric interactions driven by intuitive prompts. Therefore, addressing this gap necessitates a model that natively supports intuitive region prompting. Our proposed MedP-CLIP is designed to accurately interpret regions of interest specified via intuitive clinician prompts or perception models, establishing native region-level perception within the CLIP for medical scenarios at minimal modification cost.

3. Method

MedP-CLIP enhances the CLIP framework into a region-promptable VLM. While preserving CLIP’s inherent strength in global image-text alignment, MedP-CLIP uniquely enables region-specific visual reasoning conditioned on fine-grained spatial prompts, addressing the critical need for Medical RoIs analysis. To achieve this, we introduce two key components: (1) Large-scale Medical Region-Text Dataset: We compile the largest medical region-text dataset containing 97.3M pairs of localized image regions and corresponding textual descriptions. (2) Plug-in Prompt Modulation: We devise a lightweight, plug-in prompt modulation mechanism that seamlessly integrates clinician- specified spatial prompts (points, bounding boxes, or masks) into CLIP’s feature embedding space, dynamically guiding the model’s attention to target RoI regions. The CLIP vision encoder is trained end-to-end with the prompt integration module, while the proposed design preserves the original ViT backbone structure and introduces a lightweight region prompt integration mechanism at the feature level, thereby retaining broad compatibility with downstream medical applications that benefit from CLIP’s transferability.

3.1. Medical Image Region–Text Pairs Generation

As shown in Fig. 2(a), we generate medical image region-text pairs based on the IMed-361M dataset (Cheng et al., 2025). This dataset was originally designed for interactive segmentation, providing pixel-level mask annotations and category labels for 6.4M medical images. To construct region-text pairs, we first decompose each original image into 𝑛region-annotated samples, where 𝑛corresponds to the number of different anatomical structures in the image.

The core of our data generation pipeline leverages GPT-4o to synthesize clinically based textual descriptions. For each triplet {𝐼, 𝑀𝑘, 𝐶𝑘} (representing the original image, the k-th region mask, and the associated category label), we input this information into GPT-4o. The model is given constrained instructions to generate comprehensive descriptions covering both the global context and specific features of the masked region, as well as to produce different variants based on provided templates. Example descriptions include: “The contrast-enhanced CT image shows the segmented

Page 4 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration Figure 2: MedP-CLIP: data, architecture and training. From IMed-361M we convert pixel-level organ/lesion masks into region–text pairs by generating prompts (point/box/mask) and GPT-4o based clinical descriptions (a). Dataset statistics across modalities, anatomies and mask ratios (b). Model pipeline: the image is encoded by a ViT, while the prompt is embedded by a lightweight Prompt Encoder (c). A feature-level Attention Block fuses dense and sparse prompt embeddings with image tokens via cross/self attention (d).

liver parenchyma, with prominent vascular enhancement within the masked region” and “The grayscale abdominal scan displays the oval-shaped splenic parenchyma highlighted by the overlaid mask”. Comprehensive implementation details of the text generation framework and our quality control mechanisms are elaborated in App. A.

As shown in Fig. 2(b), we ultimately obtained a dataset of 97.3M region-text pairs, covering 15 different modalities, significantly surpassing the 6 modalities used in UniMed-CLIP pre-training. The region-text pairs were systematically categorized into six major anatomical groups (abdomen, bone, head neck, pelvis, thorax, and lesions), comprehensively covering the primary human anatomical systems. Crucially, quantitative analysis revealed that 86.9% of the regions accounted for less than 2% of the total image area, empirically demonstrating the prevalence of highly localized RoIs in medical imaging.

3.2. Medp-Clip

MedP-CLIP bridges the gap between user-provided spatial prompts and medical image understanding through a unified framework that processes visual data (𝐼), regional text descriptions (𝑇), and prompts (𝑃). As depicted in Fig. 2(c), our architecture enhances CLIP’s foundational capabilities with explicit region-aware reasoning via an attention block that jointly reasons over visual features and prompt semantics. This enables precise alignment between clinician-specified regions and textual descriptions.

3.2.1. Image And Text Encoder

We adopt CLIP’s Vision Transformer (ViT) architecture as the image encoder, specifically instantiated as ViT- with the prompt integration module, while the CLIP text encoder is kept frozen. The vision encoder processes the input image 𝐼into a sequence of patch tokens 𝐅𝐈∈ℝ𝐿×𝐷1, where 𝐿denotes the number of patches and 𝐷1 represents the hidden feature dimension. Meanwhile, the CLIP text encoder encodes the regional text description 𝑇, producing a global text feature 𝐅𝑇∈ℝ𝐷, where 𝐷represents the feature dimension.

Page 5 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration

3.2.2. Prompt Encoder

Inspired by SAM (Kirillov et al., 2023), our prompt encoder processes two types of prompt information: sparse prompts (points, boxes) and dense prompts (masks). For sparse prompts, we combine positional encoding and learnable type embeddings to represent each point or box, forming sparse prompt features 𝐅𝐒𝐩𝐚𝐫𝐬𝐞∈ℝ𝑆×𝐷1, where S denotes the number of sparse prompts. For dense prompts, we employ a set of lightweight convolutional networks for feature embedding, generating dense prompt features 𝐅𝐃𝐞𝐧𝐬𝐞∈ℝ𝐿×𝐷1 that maintain spatial resolution consistency with the image patch tokens.

3.2.3. Attention Block

As shown in Fig. 2(d), this block is the core component of MedP-CLIP, leveraging the geometric positions and type information of prompts to guide the model in focusing on user-specified image regions. It primarily consists of the following sequential processing steps: (1) Self-attention computation for sparse embeddings, and element-wise addition of dense embeddings (if present) with image embeddings; (2) Use sparse embeddings as queries to perform cross-attention computation with image embeddings, followed by a pointwise MLP update for each embedding; (3) Use image embeddings as queries to perform cross-attention computation with sparse embeddings; (4) Obtain the final image embedding 𝐅𝐀∈ℝ𝐷through 𝐺𝐴𝑃and 𝑀𝐿𝑃. During computation, each self-attention/cross-attention module and 𝑀𝐿𝑃includes residual connections and layer normalization layers. The above process can be expressed

Given A Batch Of 𝑁triplets {𝐼𝑖, 𝑃𝑖, 𝑇𝑖}𝑁

𝑖=1 where each triplet consists of an image, a user prompt, and a region- specific text description, we first normalize the output features from the attention module 𝐅𝐀and the text encoder 𝐅𝐓 to obtain normalized embeddingŝ 𝐅𝐀and̂ 𝐅𝐓. We then compute the following objectives including similarity matrix calculation, bidirectional contrastive losses, and the final symmetric loss function:

(2)

where 𝜏is a learnable temperature. It should be noted that while our ground-truth data is derived from precise pixel- level masks, we actively simulate degraded, sparse inputs during training to decouple the model from high-precision mask dependency. Specifically, to prevent the model from over-relying on any single prompt type while maintaining holistic recognition capabilities, for each image we randomly select one prompt type from point prompts, box prompts, mask prompts, and point+box combined prompts with probabilities {0.3, 0.3, 0.3, 0.1}. This strategy forces the model to map imprecise spatial inputs to complex semantic regions, thereby ensuring robust native promptability during inference. Additionally, with a small probability (𝑝= 0.1), prompts are entirely discarded to train on complete image- text pairs.

3.2.5. Medp-Clip For Downstream Tasks

Following pre-training, MedP-CLIP enables versatile downstream capabilities: (1) image-level zero-shot recog- nition with optional prompt customization; (2) region-level retrieval and classification within medical images using diverse prompt formats (points, bounding boxes, or combinations); (3) serving as the backbone for robust interactive

Page 6 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration Table 1. Hyperparameter configurations for MedP-CLIP pre-training

Augmentation

RandScaleIntensityd, RandShiftIntensityd (Cardoso et al., 2022)

Backbones

medical image segmentation when integrated with the Segment Anything Model (SAM); and (4) plug-and-play replacement of existing visual encoders within medical large language model pipelines, requiring no architectural modifications.

4.1. Training Setup

All MedP-CLIP variants were trained on 16 NVIDIA RTX 4090 GPUs (24GB). The hyperparameters listed in Tab. 1 were consistently applied across all pre-training experiments.

4.2. Evaluation Dataset And Metrics

To rigorously evaluate MedP-CLIP across the four downstream tasks, including zero-shot image-level classifi- cation, region-level retrieval, interactive segmentation, and medical visual question answering (VQA), we conduct experiments on a diverse and comprehensive benchmark comprising 11 publicly available medical imaging datasets spanning 8 modalities (CT, MRI, fundus, X-ray, etc.) and 129,298 expert-annotated samples. As shown in Tab. 2, all evaluation datasets are carefully held out from the pre-training corpus to ensure a strict assessment of out-of-distribution generalization under realistic clinical conditions. For AMOS2022 (Ji et al., 2022), we explicitly separated the dataset at the patient level before region-text pair construction. Cases assigned to downstream evaluation were completely excluded from the GPT-4o-based text generation process and from the MedP-CLIP pre-training corpus. Therefore, no patient, image, mask, or region-text pair from the AMOS2022 evaluation subset overlaps with the pre-training data, preventing patient-level and image-level leakage. Specifically, for zero-shot classification, we use COVID-CT (Yang et al., 2020), ACL (Bien et al., 2018), ChestCT (Matsuyama et al., 2024), and ACRIMA (Ovreiu et al., 2021). Region- level retrieval is evaluated on Br35H (Wageh et al., 2024), AMOS2022 (Ji et al., 2022), and 3D-IRCADB (Soler et al., 2010). Interactive segmentation is benchmarked on ISLES (Hernandez Petzsche et al., 2022), SegThor (Lambert et al., 2020), and TotalSegmentator MRI (Akinci D’Antonoli et al., 2025). For medical VQA, we adopt MeCoVQA- R (Huang et al., 2025), the most comprehensive region-aware benchmark to date, featuring 126,661 training and 2,637 test questions across eight modalities.

Evaluation metrics are aligned with task semantics and community standards: Top-1/Top-5 accuracy and recall for classification and region retrieval; Dice coefficient for interactive segmentation; and precision and recall for region-level VQA, following the MedPLIB protocol (Huang et al., 2025). This unified evaluation framework enables a holistic and clinically meaningful assessment of MedP-CLIP’s multimodal reasoning, spatial grounding, and zero-shot adaptability across heterogeneous medical imaging scenarios.

4.3. Image-Level Zero-Shot Classification

As mentioned above, we conducted zero-shot classification experiments on four different medical imaging datasets: COVID-CT (Yang et al., 2020), ACL (Bien et al., 2018), ChestCT (Matsuyama et al., 2024), and ACRIMA

Page 7 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration Table 2. Overview of evaluation datasets for image-level classification, region-level retrieval, and interactive segmentation

Evaluation Type

Image Num. Mask Num.

2

Image-Level Class.

2

Image-Level Class.

4

Image-Level Class.

2

Image-Level Class.

2

Region-Level Class.

15

Region-Level Class.

16

Region-Level Class.

2

Interactive Seg.

4

Interactive Seg.

12

Interactive Seg.

8100

Table 3. Comparison of zero-shot image-level classification accuracy across four medical datasets

54.35%

Figure 3: Qualitative comparison of attention maps for zero-shot classification on COVID-CT and ACL datasets. (Ovreiu et al., 2021). Among them, ChestCT contains four categories (normal, adenocarcinoma, large cell carcinoma, and squamous cell carcinoma), while the other datasets are binary classification tasks. Although MedP-CLIP is primarily designed for region-level reasoning, our results demonstrate that it effectively retains the global representation capabilities inherent to CLIP and its recent medical variants. As shown in Tab. 3, MedP-CLIP achieves the best performance on the COVID-CT and ACL datasets, with improvements of up to 3.52% compared to the best existing methods. The model also maintains strong performance on the more challenging ChestCT dataset, which includes multiple disease categories that may benefit from our region-aware design. Notably, MedP-CLIP achieves an

Page 8 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration average accuracy of 54.35% across all datasets, representing a 1.72% improvement over the current top-performing GenMedCLIP baseline.

To further corroborate these quantitative findings, we provide qualitative visual comparisons in Fig. 3. As illustrated, state-of-the-art global representation models, such as Biomed-CLIP and BMC-CLIP, often exhibit diffuse attention maps and struggle to localize spatially confined pathological regions. While MedP-CLIP without explicit prompts already demonstrates improved focal attention (e.g., yielding a 77.26% confidence score on the COVID- CT sample compared to Biomed-CLIP’s 11.09%), the introduction of point prompts enables highly precise region- text alignment. By explicitly guiding the model to the target lesion, MedP-CLIP successfully mitigates irrelevant background interference, resulting in the most accurate attention maps and the highest diagnostic confidence scores.

These results confirm that our method, while explicitly learning to handle regional prompts, does not compromise overall semantic alignment, making MedP-CLIP more versatile and reliable in practical clinical applications. Table 4. Region-level classification performance on the Br35H dataset under different prompt settings. Accuracy values of MedP-CLIP with prompts are reported as mean ± std over five runs, with the corresponding 95% confidence intervals shown in brackets. Params. and FPS denote the number of model parameters and inference speed, respectively

Data Scale

Params.

Dataset

Reg. Info.

4.4. Region-Level Retrieval And Classification

In addition to global classification, we further evaluated region-level understanding on Br35H, a brain tumor classification benchmark that requires localized diagnostic reasoning. As shown in Tab. 4, MedP-CLIP without spatial prompts achieves 58.61% accuracy, indicating that image-level representations alone are insufficient for accurately identifying localized pathological regions. Introducing lightweight spatial prompts substantially improves performance: the point prompt increases the accuracy to 69.79% ± 0.14%, while the bounding box prompt further raises it to 79.37% ± 0.21%, corresponding to a 20.76 percentage-point gain over the prompt-free setting. This result also surpasses the recent region-aware baseline GenMedCLIP by 0.44 percentage points. Interestingly, combining point and box prompts with the ViT-B/16 backbone yields 78.93% ± 0.30%, which is slightly lower than using the box prompt alone, suggesting that the bounding box already provides sufficiently precise regional constraints for this dataset and that additional point information may not always bring complementary benefits. The MedP-CLIP ViT-B/16 variants contain 167M parameters and run at 269 FPS when using prompts, which is comparable to CLIP, MedCLIP, BiomedCLIP, and UniMed-CLIP, while being substantially faster than CARZero and BMC-CLIP. This indicates that the performance gain is not obtained at the cost of heavy computational overhead. When scaling to the ViT-L/14 backbone, MedP-CLIP achieves the best accuracy of 92.45% ± 0.11%, outperforming strong medical vision-language baselines such as BiomedCLIP and BMC-CLIP by 0.50 and 0.75 percentage points, respectively. Moreover, the 95% confidence interval of MedP-CLIP (ViT-L/14, w. both) is [92.31, 92.59], whose lower bound remains higher than the accuracies of the strongest competing baselines, including BiomedCLIP and BMC-CLIP. This indicates that the observed improvement is stable across repeated runs rather than being caused by random variation. Although this larger

Page 9 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration variant introduces higher computational cost, with 455M parameters and 71 FPS, it provides a favorable high-accuracy configuration for scenarios where diagnostic performance is prioritized. The consistently small standard deviations across prompt-based variants further indicate stable performance over repeated runs.

Figure 4: Performance comparison of region-level classification on multi-organ datasets. “Δ” denotes the performance difference between Top-5 and Top-1. “Rect.” indicates rectangular region prompts generated from original masks via Alpha-CLIP.

We extended the evaluation to multi-organ matching tasks on AMOS2022 and 3D-IRCADB. Due to the lack of region-level tuning in medical baseline models, we benchmarked against Alpha-CLIP, a natural image foundation model. As shown in Fig. 4, MedP-CLIP consistently outperformed Alpha-CLIP in region-aware tasks. Bounding box prompts, which provide explicit spatial context, generally yielded higher accuracy than point prompts, while composite prompts achieved the best overall accuracy and recall. To further examine the category-wise behavior, we provide a per-organ radar comparison on the 3D-IRCADB dataset.

Figure 5: Comparison of per-category Top-1 and Top-5 recall performance on the 3D-IRCADB dataset. As shown in Fig 5, MedP-CLIP maintains consistently strong recall across most anatomical categories, including artery, bladder, heart, kidney, lung, adrenal gland, pancreas, spleen, and small intestine, where the performance is close

Page 10 Of 23

MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration Figure 6: Visual validation of MedP-CLIP’s region-aware retrieval capabilities. Table 5. Interactive segmentation results on external datasets

Dataset / Dice (%)

ISLES SegThor Totalseg.

70.36

to or reaches the outer boundary of the chart. In contrast, Alpha-CLIP exhibits highly uneven behavior across organs, with substantially degraded recall on several clinically important structures such as artery, biliary system, adrenal gland, veins, and small intestine. This clear per-class gap suggests that the advantage of MedP-CLIP does not merely come from improvements on a few dominant categories, but from more stable region-aware recognition across heterogeneous anatomical structures. Moreover, the comparison between point and box prompts reveals complementary behaviors: box prompts provide more reliable localization for spatially ambiguous or small structures, whereas point prompts can achieve comparable or slightly better performance for organs with clear visual boundaries. These fine-grained results further support the effectiveness of our region-aware prompt integration mechanism in handling diverse anatomical targets. Fig. 6 visually validates these findings: under flexible regional prompts, MedP-CLIP was able to retrieve semantically aligned text descriptions with near-certain confidence (>98%). These findings confirm that: (i) our feature-level prompt fusion effectively integrates positional guidance; and (ii) prompt quality is directly correlated with discriminative localization capability, which is a key advantage for medical image interpretation.

4.5. Medp-Clip In Interactive Segmentation

Leveraging MedP-CLIP’s intrinsic prompt affinity, we developed MedP-CLIP+SAM by substituting the original SAM ViT encoder. This architecture exploits synergy through shared prompt encoding features between MedP- CLIP (focusing the backbone on relevant regions) and SAM’s mask decoder (guiding segmentation). Following the IMIS-Net (Cheng et al., 2025) protocol, we fine-tuned the decoder (<1% of encoder parameters) for 5 epochs on IMed-361M, achieving rapid convergence. As detailed in Tab. 5, MedP-CLIP+SAM delivers consistent and substantial improvements over the original SAM across all datasets. Critically, our model achieves new state-of- the-art performance on ISLES (73.14%) and TotalSegmentator MRI (81.55%), surpassing IMIS-Net. These results underscore MedP-CLIP’s dual capability: it serves as a powerful backbone for downstream medical vision tasks while uniquely enabling targeted performance gains by directing computational resources towards clinically critical regions.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.