Complete Technical Paper Content — Ai Radiology Projects
1. Executive Abstract, Problem Formulation & Background
MGH Radiology Llama: A Llama 3 70B Model for Radiology Yucheng Shi Affiliation: School of Computing, University of Georgia Peng Shu Affiliation: School of Computing, University of Georgia Zhengliang Liu Affiliation: School of Computing, University of Georgia Zihao Wu Affiliation: School of Computing, University of Georgia Quanzheng Li Affiliation: Department of Radiology, Massachusetts General Hospital and Harvard Medical School Xiang Li Affiliation: Department of Radiology, Massachusetts General Hospital and Harvard Medical School Abstract In recent years, the field of radiology has increasingly harnessed the power of artificial intelligence (AI) to enhance diagnostic accuracy, streamline workflows, and improve patient care. Large language models (LLMs) have emerged as particularly promising tools, offering significant potential in assisting radiologists with report generation, clinical decision support, and patient communication. This paper presents an advanced radiology-focused large language model: MGH Radiology Llama. It is developed using the Llama 3 70B model, building upon previous domain-specific models like Radiology-GPT and Radiology-Llama2. Leveraging a unique and comprehensive dataset from Massachusetts General Hospital, comprising over 6.5 million de-identified medical reports across various imaging modalities, the model demonstrates significant improvements in generating accurate and clinically relevant radiology impressions given the corresponding findings. Our evaluation, incorporating both traditional metrics and a GPT-4-based assessment, highlights the enhanced performance of this work over general-purpose LLMs. 1 Introduction The field of radiology has been increasingly leveraging artificial intelligence to enhance diagnostic accuracy, streamline workflows, and improve patient care. Large language models (LLMs) have shown particular promise in this domain, offering the potential to assist radiologists in report generation, clinical decision support, and patient communication [ 1 , 2 , 3 , 4 ] . Previous work such as Radiology-GPT [ 3 ] and Radiology-Llama2 [ 5 ] demonstrated the effectiveness of domain-specific LLMs in radiology tasks, outperforming general-purpose models in generating coherent and clinically relevant impressions from radiological findings. Building upon these foundations, we present an advanced iteration of radiology-focused LLM, which we have developed using the Llama 3 70B [ 6 ] model as its base. This new model represents a significant leap forward in scale and capability, addressing key challenges in applying LLMs to the radiology domain: • Enhanced Model Scale: By utilizing the Llama 3 70B model, we significantly increase the capacity and potential performance of our system compared to previous 7B-based models. This larger model allows for more nuanced understanding and generation of radiology-specific language. • Unique and Comprehensive Dataset: We leverage an extensive and diverse dataset from Massachusetts General Hospital (MGH), comprising over 6.5 million medical reports spanning a decade (2008-2018). This dataset is unique in its scale and breadth, encompassing a wide range of imaging modalities (including CT, MRI, X-ray, and Fluoroscopic imaging) and body regions (such as abdomen, cardiac, chest, and limbs). The richness and diversity of this dataset provide an unparalleled foundation for training a versatile radiology AI assistant. • Privacy and Compliance: By utilizing a locally deployable model trained on de-identified data, we maintain strict adherence to patient privacy regulations, a critical consideration in healthcare AI applications [ 7 , 8 , 9 , 10 ] . • Comprehensive Evaluation: Our assessment incorporates traditional metrics such as ROUGE [ 11 ] and BERTScore [ 12 ] , as well as a novel GPT-4 based evaluation, providing a multi-faceted analysis of the model’s capabilities. This paper details our methodology in preprocessing the MGH dataset, the fine-tuning process, and the evaluation of the resulting model. We demonstrate significant improvements over both general-purpose LLMs and our previous iterations in generating accurate, concise, and clinically useful radiology impressions across a wide range of imaging modalities and anatomical regions. Our work contributes to the growing body of research on specialized medical AI, showcasing the potential of large-scale, domain-specific language models in radiology. By developing a model that can effectively process and generate radiology reports for various imaging techniques and body parts, we aim to provide a versatile tool that can support radiologists in their daily practice, potentially improving diagnostic accuracy and efficiency. 2 Related work 2.1 Large Language Models in Healthcare General-purpose LLMs such as GPT-3 [ 13 ] , GPT-4 [ 14 ] , and PaLM [ 15 ] have demonstrated impressive capabilities across various domains, including healthcare. These models have shown potential in medical question answering, clinical decision support, and even assisting in medical diagnosis. Their ability to process and generate human-like text has opened up new possibilities for automating various aspects of healthcare, from patient communication to preliminary analysis of medical literature. The versatility of these models allows them to adapt to a wide range of medical topics, potentially serving as powerful tools for healthcare professionals and researchers alike. However, the application of general-purpose LLMs in specialized medical fields faces significant limitations. These models often lack the depth of domain-specific knowledge required in areas like radiology, potentially leading to inaccuracies or misinterpretations. Privacy concerns are paramount, as using these models typically involves sending sensitive medical data through external APIs, conflicting with strict patient confidentiality requirements. Moreover, these LLMs may not capture the nuanced language and reporting style used by medical professionals, particularly in specialized fields. The need for on-premise solutions in many healthcare institutions further complicates the deployment of these large models. These limitations have spurred the development of specialized models for healthcare applications, designed to address these challenges while leveraging the power of large language models. 2.2 Domain-Specific Language Models in Radiology Several attempts have been made to create domain-specific language models for radiology. RadBERT [ 16 ] and ClinicalBERT [ 17 ] are examples of BERT-based models fine-tuned on radiology reports. These models showed improvements in tasks such as named entity recognition and report classification but were limited in their ability to generate coherent, full-length reports. Radiology-GPT, built upon the Alpaca-7B model [ 18 ] , demonstrated the potential of instruction-tuned models in radiology. This was followed by Radiology-Llama2, which utilized the Llama2 architecture to achieve state-of-the-art performance on radiology report generation tasks. Other notable works in this area include XrayGLM [ 19 ] , which focused on chest X-ray report generation, and DoctorGLM [ 20 ] , a more general medical LLM with capabilities in radiology. In our current work, we focus on leveraging larger and more advanced base models. We use the latest open-source LLM, Llama 3, with 70 billion parameters, which represents a significant increase in model size compared to the 7B versions used in many previous works. This increase in scale potentially allows for more nuanced understanding and generation of radiology-specific language. Additionally, we introduce an advanced evaluation approach to provide a more comprehensive assessment of model performance. We use GPT-4, an advanced LLM, as a judge to complement traditional automatic scoring methods such as the ROUGE family of scores and BERTScore. This multi-faceted evaluation strategy aims to combine the strengths of automated metrics with the nuanced understanding and explainable capabilities. 3 Methodology Our method to build this model involves two main steps: Preprocessing of Massachusetts General Hospital (MGH) dataset and instruction fine-tuning. For fine-tuning, we utilize both fully fine-tuning and QLoRA fine-tuning [ 21 ] on Llama 3 70B model. 3.1 Dataset and preprocessing We utilize abundant data from Massachusetts General Hospital where it contents over 6,500,000 medical reports from patients who were admitted to MGH between 2008 and 2018. These comprehensive medical reports encompass a wide range of imaging modalities and cover various body parts. The imaging modalities include CT (Computed Tomography), MRI (Magnetic Resonance Imaging), X-ray, and Fluoroscopic imaging. The reports provide detailed examinations of different body regions such as the abdomen, cardiac (heart), chest, limbs, and more. We focus on the radiology reports since they contain plenty of information such as radiologists’ interpretations for corresponding medical images. We leverage the exam codes, original reports and impressions to establish our dataset. The exam codes indicate the modality of corresponding images of the original reports. Original reports contain the findings that specify the detailed observation from radiology images. And impressions part summarizes these detailed observation. To build our dataset, we extract findings from the original reports and impressions to form pairs for instruction fine-tuning. Given an instruction with findings as input, the model should generate desired impression as output. For instruction, we follow [ 22 ] to create 20 different instructions asking for impressions. We randomly select one instruction for each pair. We also combine the exam code with finding in the input to provide richer information. Similarly, in the output impression, we put a prepend phrase selected randomly from 10 sentences list with a certain probability. Since these radiology reports come from MGH, which means they contain sensitive privacy information such as patient names and doctor names, we remove all the names in the reports by using regular expression to match the sentences that includes names. Then, we remove the unqualified reports that would cause potential hallucinations. They can be summarized as: (1) reports that don’t have either findings or impressions, (2) reports with findings less than 10 words. Besides, we remove the extra white space to avoid format problems. Finally, we obtain 4,354,321 reports as training data. For testing data, we implement same operation for 2114 new reports apart from training data. An example of generated impression from MGH Radiology Llama is shown in figure 1 . 3.2 Fine Tuning Methods Figure 1 : The upper part shows the User question with findings. The bottom part is impression from MGH Radiology Llama. The ground truth impression is also provided. When fine-tuning large language models like the Llama 3 70B model, two primary methods are commonly employed: fully fine-tune and LoRA fine-tune. Fully fine-tune involves adjusting all the parameters of the pre-trained model on a specific downstream task. This method is powerful as it allows the model to fully adapt to the new task, but it requires substantial computational resources and large amounts of data, which can be costly and time-consuming. On the other hand, LoRA fine-tune is a more efficient approach that focuses on learning low-rank updates to the model’s weights, rather than modifying all the parameters. This method introduces additional trainable matrices with much fewer parameters than the original model, significantly reducing the computational burden while maintaining high performance. LoRA fine-tune is particularly useful when resources are limited or when fine-tune needs to be done quickly, making it a popular choice for adapting large models like Llama 3 70B to specific tasks with fewer data and compute requirements. QLoRA allows for further computation resources saving from LoRA. QLoRA usually first quantize the model weights to a lower precision, typically 4-bit, and then apply low-rank adaptation. This approach significantly reduces the memory footprint of the model, enabling the fine-tuning of models with billions of parameters on hardware with very limited memory, such as a single GPU. Despite the reduction in precision and the use of low-rank matrices, QLoRA maintains competitive performance, making it a powerful method for adapting large-scale pre-trained models to specific tasks efficiently. In our experiments, we implement both fully fine-tune and QLoRA methods since we have enormous data as well as enough computation resources to support fully fine tune for Llama 3 70B. We also want to explore the difference between fully fine-tune and Lora fine-tune because there is no such comparison results on radiology reports foundation model. After this project we can have a clear guideline for future fine-tune method selection. 4 Experiments In this section, we describe our experimental setup for training and evaluating the MGH Radiology Llama, a large language model specialized for radiology. Our experiments encompass model preparation, dataset handling, and the training process, all optimized for high-performance computing infrastructure. 4.1 Model Preparation We based our model on the Llama 3 architecture, specifically the meta-llama/Meta-Llama-3-70B-Instruct variant [ 6 ] . To optimize for both performance and efficiency, we applied 4-bit quantization using BitsAndBytesConfig [ 23 ] . This quantization technique significantly reduced the model’s memory footprint, allowing us to work with larger models within our GPU constraints. We further enhanced the model’s adaptability by implementing Low-Rank Adaptation (LoRA) [ 24 ] , configured using PeftConfig. LoRA enabled us to fine-tune the model efficiently by adapting a small subset of parameters while maintaining overall performance. 4.2 Dataset Handling Our dataset processing pipeline was designed to maximize efficiency and GPU utilization. We began by loading our dataset from a JSON file, which was converted from raw CSV files containing radiology reports and associated metadata. To optimize the tokenization process, we pre-tokenized the text using the LLama 3 tokenizer and stored the results in cache, eliminating the need for repeated tokenization during training. A key innovation in our approach was the implementation of sequence packing [ 25 ] . This technique allowed us to concatenate multiple sequences up to a maximum length of 2048 tokens, significantly improving GPU utilization. We carefully managed attention masks to ensure the model could differentiate between packed sequences. The final step in our data preparation involved splitting the dataset into training and evaluation sets, with 99.9% allocated for training and 0.1% reserved for evaluation. 4.3 Training Process For the training phase, we employed the SFTTrainer for supervised fine-tuning, incorporating several advanced techniques to optimize performance and resource utilization. We enabled gradient checkpointing to balance memory efficiency with computational demands, allowing us to train larger models than would otherwise be possible given our memory constraints [ 26 ] . We leveraged DeepSpeed ZeRO Stage 3 for distributed training across multiple GPUs [ 27 ] . This approach sharded model parameters, gradients, and optimizer states across our GPU cluster, enabling us to train substantially larger models than a single GPU could handle. Our training process also utilized mixed precision training with BF16 (Brain Float 16), chosen for its wider dynamic range and reduced risk of numerical issues compared to FP16 [ 28 ] . To further optimize our training, we implemented gradient accumulation, effectively increasing our batch size without proportionally increasing memory usage. This technique allowed us to simulate larger batch sizes, improving training stability and potentially accelerating convergence [ 29 ] . 4.4 Hardware Utilization Our experiments were conducted on a high-performance computing cluster equipped with 8 NVIDIA H100 GPUs. This advanced hardware allowed us to fully leverage the benefits of our optimized training pipeline. The H100’s Tensor Cores were particularly beneficial for our BF16 mixed precision training, providing significant speedups in computational operations [ 30 ] . The combination of 4-bit quantization and distributed training techniques enabled us to efficiently utilize the massive parallel processing capabilities of our GPU cluster. This setup allowed us to process larger batch sizes and train on more extensive datasets than would be possible with less optimized configurations. By integrating these advanced training techniques with state-of-the-art hardware, we were able to efficiently train and fine-tune the MGH Radiology Llama model, achieving high performance while effectively managing computational resources. This approach positions our model to potentially advance the state of the art in AI-assisted radiology interpretation and reporting. 5 Evaluation Figure 2 : Prompt design for GPT-4o evaluation based on Clinician-Curated Criteria In this section, in order to have a comprehensive evaluation of our fine-tuned model and to compare with baseline Llama 3, we adapt two types of methods: Traditional metrics which focus more on low level words matching and GPT evaluation that estimates the accuracy and similarity for high semantic level. 5.1 Traditional Metrics For traditional metrics, we use Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and BERTScore [ 12 ] for the evaluations. ROUGE metric is a very common metric in natural language processing (NLP) region. It is a set of metrics used for evaluating automatic summarization and machine translation in NLP. The metrics compare the generated impressions against the reference ground truth. ROUGE metrics range between 0 and 1, with higher scores indicating higher similarity. Since ROUGE-1 and ROUGE-2 only compare overlap of unigrams (each word) and bigrams between produced impressions and reference ground truth, here we apply ROUGE-L for Longest Common Subsequence (LCS). LCS takes into account sentence-level structure similarity naturally and identifies longest co-occurring in sequence n-grams automatically. BERTScore is another popular automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence [ 12 ] . Instead of searching for exact matching, BERTScore compute the similarity in contextual embeddings. The results are shown in table 1 . Models ROUGE-L BERTScore Precision BERTScore Recall BERTScore F1 GPT-4o Score Llama 3 70B 0.1494 0.8246 0.8690 0.8460 3.65 Llama 3 70B QLoRA 0.2919 0.8682 0.8864 0.8768 4.92 Llama 3 70B Fully fine-tune 0.2890 0.8735 0.8817 0.8771 4.74 Table 1 : ROUGE-L, BERTScore and GPT-4o evaluation results for three models. 5.2 GPT Evaluation Following the approach in [ 31 ] , we utilize GPT-4o for MGH to evaluate our model’s performance. We design a prompt aimed at leveraging GPT-4o’s strong capability to compare the accuracy and similarity of generated impressions. The prompt design is shown in figure 2 . The score varies from 0-10 where higher score means the generated impression matches expected real impression better. An explanation will come after the score, indicating the criteria and reason GPT-4o assign this score. An example of GPT-4o evaluation is shown in figure 3 . It is obvious from the explanation that the evaluation of GPT-4o follows requirement in prompt. GPT-4o can distinguish the missing part and determine stylistic concordance and precision of medical terminology. We also list this experiment result in table 1 . 5.3 Experiment Results From table 1 , we can find that both of our fine-tuned Llama 3 models outperform baseline model. Our models achieve significant improvement. We get 100% promotion for ROUGE-L score, 4%-5% for BERTScore precision and more than 1 point on GPT-4o score. This means our models extract more precise radiology impressions from the findings. Comparing with baseline, our generated impressions align better with expected real conclusions derived from radiologists. From low level words matching to high level semantic accuracy, our models perform much better. We notice that fully fine-tune model is the best model but QLoRA fine-tuned model outperforms in some tests. However, the difference between QLoRA and fully fine-tune model is not obvious. We think it is because Llama 3 70B model comes with enormous parameters such that even adjusting limited amount of weights, Llama 3 70B model is still able to dig out its background knowledge in radiology and learn to expresses more precisely like a radiology expert. While in the mean time, fully fine tuning costs almost twice GPU training time. It shows that for large models, QLoRA fine-tune can reach similar performance but with much less training resources. The average GPT-4o score is low for all models. We think it is reasonable because test set contains reports from various modalities of medical images. Some of them are harder to diagnose even for experienced radiologists. We check the explanation for the cases with low score. The main reasons located in missing important findings that are shown in the ground truth. Another reason is that generated impressions include irrelevant conclusions which are not presented in expected impressions. The latter may be caused by difficult cases or hallucination. Overall, the fully fine-tune Llama 3 70B model outperforms other two models but QLoRA has very little difference with the best model. This indicates the larger the model is, the more benefits QLoRA fine-tune can obtain. Figure 3 : Prompt design for GPT-4o evaluation based on Clinician-Curated Criteria 6 Conclusion In this project we utilize a huge amount of data from MGH to leverage the potential of Llama 3 70B on radiology reports. We demonstrate the diversity of radiology reports collecting from MGH in both size and modality. We also demonstrate our de-identified data to protect patients privacy. By applying QLoRA and fully fine-tune, Llama 3 70B model can achieve much better performance in this down stream tasks. We find that model with massive parameters can benefit more from QLoRA fine-tune by obtaining similar performance but with much fewer computation resources. Our project still remains some limitations. Firstly, due to the time limit we didn’t compare our models with previous state-of-art work such as Radiology-llama2 [ 5 ] . Secondly, Meta has released the latest Llama 3.1 models. They mention the performance of new series has exceeded previous version, which means our models cannot take advantage of these powerful models. Thirdly, our model still exists hallucination. It is not qualified for automatic radiology impression generation. For the future work, we have following tasks to be continued. Firstly, we will retrain our model with advanced Llama 3.1 70B model to make full use of foundation model. Since QLoRA performs well for larger model, we will attempt to retain model with Llama 3.1 405B model. Secondly, we will apply LLM involved methods to further clean our data. It enables safer patient privacy and compliance for commercial usage. It may also decrease the hallucination. Thirdly, we will enrich evaluation methods to have more comprehensive results. Fourthly, detection of hallucination will be added in the model. We hope to decrease hallucination according to some typical examples. References [1] Sunkyu Kim, Choong-kun Lee, and Seung-seob Kim. Large language models: a guide for radiologists. Korean Journal of Radiology , 25(2):126, 2024. [2] Zhengliang Liu, Lu Zhang, Zihao Wu, Xiaowei Yu, Chao Cao, Haixing Dai, Ninghao Liu, Jun Liu, Wei Liu, Quanzheng Li, et al. Surviving chatgpt in healthcare. Frontiers in Radiology , 3:1224682, 2024. [3] Zhengliang Liu, Aoxiao Zhong, Yiwei Li, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Peng Shu, Cheng Chen, Sekeun Kim, Haixing Dai, Lin Zhao, Lichao Sun, Dajiang Zhu, Jun Liu, Wei Liu, Dinggang Shen, Xiang Li, Quanzheng Li, and Tianming Liu. Radiology-gpt: A large language model for radiology, 2024. [4] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology , page 100017, 2023. [5] Zhengliang Liu, Yiwei Li, Peng Shu, Aoxiao Zhong, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Jie Luo, Cheng Chen, et al. Radiology-llama2: Best-in-class large language model for radiology. arXiv preprint arXiv:2309.06419 , 2023. [6] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. [7] Steven M Williamson and Victor Prybutok. Balancing privacy and progress: a review of privacy challenges, systemic oversight, and patient perceptions in ai-driven healthcare. Applied Sciences , 14(2):675, 2024. [8] Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, et al. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032 , 2023. [9] Sakib Shahriar, Sonal Allana, Seyed Mehdi Hazratifard, and Rozita Dara. A survey of privacy risks and mitigation strategies in the artificial intelligence life cycle. IEEE Access , 11:61829–61854, 2023. [10] Zhengliang Liu, Aoxiao Zhong, Yiwei Li, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Peng Shu, Cheng Chen, Sekeun Kim, et al. Tailoring large language models to radiology: A preliminary approach to llm adaptation for a highly specialized domain. In International Workshop on Machine Learning in Medical Imaging , pages 464–473. Springer, 2023. [11] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. [12] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 , 2019. [13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020. [14] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. [15] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1–113, 2023. [16] An Yan, Julian McAuley, Xing Lu, Jiang Du, Eric Y Chang, Amilcare Gentili, and Chun-Nan Hsu. Radbert: adapting transformer-based language models to radiology. Radiology: Artificial Intelligence , 4(4):e210258, 2022. [17] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342 , 2019. [18] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023. [19] Yaofei Duan Rongsheng Wang, Patrick Pang Junrong Li, and Tao Tan. Xrayglm: The first chinese medical multimodal model that chest radiographs summarization. https://github.com/WangRongsheng/XrayGLM , 2023. [20] Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Linlin Huang, Qian Wang, and Dinggang Shen. Doctorglm: Fine-tuning your chinese doctor is not a herculean task. arXiv preprint arXiv:2304.01097 , 2023. [21] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36, 2024. [22] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024. [23] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861 , 2021. [24] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. [25] Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027 , 2021. [26] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016. [27] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1–16. IEEE, 2020. [28] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740 , 2017. [29] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. arXiv preprint arXiv:1806.00187 , 2018. [30] NVIDIA Corporation. Nvidia h100 tensor core gpu architecture. Technical report, NVIDIA, 2022. [31] Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical visual instruction tuning with clinician preference alignment. arXiv preprint arXiv:2406.13173 , 2024.
2. Related Work, Literature Survey & Algorithmic Foundations
In recent years, the field of radiology has increasingly harnessed the power of artificial intelligence (AI) to enhance diagnostic accuracy, streamline workflows, and improve patient care. Large language models (LLMs) have emerged as particularly promising tools, offering significant potential in assisting radiologists with report generation, clinical decision support, and patient communication. This paper presents an advanced radiology-focused large language model: MGH Radiology Llama. It is developed using the Llama 3 70B model, building upon previous domain-specific models like Radiology-GPT and Radiology-Llama2. Leveraging a unique and comprehensive dataset from Massachusetts General Hospital, comprising over 6.5 million de-identified medical reports across various imaging modalities, the model demonstrates significant improvements in generating accurate and clinically relevant radiology impressions given the corresponding findings. Our evaluation, incorporating both traditional metrics and a GPT-4-based assessment, highlights the enhanced performance of this work over general-purpose LLMs.
3. Dataset Ingestion, Preprocessing & Stochastic Augmentations
The field of radiology has been increasingly leveraging artificial intelligence to enhance diagnostic accuracy, streamline workflows, and improve patient care. Large language models (LLMs) have shown particular promise in this domain, offering the potential to assist radiologists in report generation, clinical decision support, and patient communication [ 1 , 2 , 3 , 4 ] . Previous work such as Radiology-GPT [ 3 ] and Radiology-Llama2 [ 5 ] demonstrated the effectiveness of domain-specific LLMs in radiology tasks, outperforming general-purpose models in generating coherent and clinically relevant impressions from radiological findings.
4. Neural Network Backbone, Layer Specifications & Loss Formulations
Radiology-GPT: A Large Language Model for Radiology Zhengliang Liu Affiliation: School of Computing, University of Georgia Aoxiao Zhong Affiliation: Department of Electrical Engineering, Harvard University Yiwei Li Affiliation: School of Computing, University of Georgia Longtao Yang Affiliation: Department of Radiology, Second Xiangya Hospital Chao Ju Affiliation: Department of Radiology, Second Xiangya Hospital Zihao Wu Affiliation: School of Computing, University of Georgia Chong Ma Affiliation: School of Automation, Northwestern Polytechnical University Peng Shu Affiliation: School of Computing, University of Georgia Cheng Chen Affiliation: Department of Radiology, Massachusetts General Hospital and Harvard Medical School Sekeun Kim Affiliation: Department of Radiology, Massachusetts General Hospital and Harvard Medical School Haixing Dai Affiliation: School of Computing, University of Georgia Lin Zhao Affiliation: School of Computing, University of Georgia Lichao Sun Affiliation: Department of Computer Science and Engineering, Lehigh University Dajiang Zhu Affiliation: Department of Computer Science and Engineering, University of Texas at Arlington Jun Liu Affiliation: Department of Radiology, Second Xiangya Hospital Wei Liu Affiliation: Department of Radiation Oncology, Mayo Clinic Dinggang Shen Affiliation: School of Biomedical Engineering, ShanghaiTech University Affiliation: Shanghai United Imaging Intelligence Co., Ltd. Affiliation: Shanghai Clinical Research and Trial Center Xiang Li Affiliation: Department of Radiology, Massachusetts General Hospital and Harvard Medical School Quanzheng Li Affiliation: Department of Radiology, Massachusetts General Hospital and Harvard Medical School Tianming Liu Affiliation: School of Computing, University of Georgia Abstract We introduce Radiology-GPT, a large language model for radiology. Using an instruction tuning approach on an extensive dataset of radiology domain knowledge, Radiology-GPT demonstrates superior performance compared to general language models such as StableLM, Dolly and LLaMA. It exhibits significant versatility in radiological diagnosis, research, and communication. This work serves as a catalyst for future developments in clinical NLP. The successful implementation of Radiology-GPT is indicative of the potential of localizing generative large language models, specifically tailored for distinctive medical specialties, while ensuring adherence to privacy standards such as HIPAA. The prospect of developing individualized, large-scale language models that cater to specific needs of various hospitals presents a promising direction. The fusion of conversational competence and domain-specific knowledge in these models is set to foster future development in healthcare AI. A demo of Radiology-GPT is available at https://huggingface.co/spaces/allen-eric/radiology-gpt . 1 Introduction The recent rise of large language models (LLMs) such as ChatGPT [ 1 ] and GPT-4 [ 2 ] has brought about a significant transformation in the field of natural language processing (NLP) [ 3 , 1 ] . These models demonstrate unprecedented abilities and versatility, a clear progression from preceding models such as BERT [ 4 ] , and have engendered broad advancements in diverse domains [ 1 , 5 , 6 ] . Among the fields that can significantly benefit from these advancements is radiology. Radiology, by its very nature, generates a vast amount of textual data, including radiology reports, clinical notes, annotations associated with medical imaging, and more [ 7 ] . Examples of such texts include radiographic findings, annotations for Computerized Tomography (CT) scans, and Magnetic Resonance Imaging (MRI) reports, all of which require sophisticated understanding and interpretation. Despite the transformative potential, the application of LLMs in the radiology domain has been limited [ 8 , 7 , 9 ] . Large commercial models such as GPT-4 and PaLM-2 [ 10 ] , while powerful, are not readily applicable in clinical settings. HIPAA regulations, privacy concerns, and the necessity for IRB approvals pose substantial barriers [ 11 ] , primarily because these models necessitate the uploading of patient data to externally hosted platforms. This situation underscores the pressing need for a localized foundational model, specifically designed for radiology, that can work effectively within the regulatory boundaries while capitalizing on the potential of LLMs. We address this gap through the development of Radiology-GPT, an LLM specifically designed for the radiology domain. A crucial advantage of generative large language models such as Radiology-GPT, particularly in comparison to domain-specific BERT-based models such as RadBERT [ 12 ] (for radiology) or ClinicalRadioBERT [ 13 ] (for radiation oncology), lies in their flexibility and dynamism. Unlike their BERT-based counterparts, generative LLMs are not rigidly dependent on a specific input structure, allowing for a more diverse range of inputs. Additionally, they are capable of generating diverse outputs, making them suitable for tasks that were previously considered impractical, such as reasoning [ 14 , 15 ] . This further extends their versatility. This inherent conversational capability of generative LLMs [ 1 ] positions them as invaluable aids to medical professionals, including radiologists. They can provide contextual insights and responses in a conversational manner, mimicking a human-like interaction, and hence enhancing the usability of these models in a clinical setting. Furthermore, these models eliminate the need for complex fine-tuning processes and the associated labor-intensive manual annotation procedures. This attribute reduces both the development time and cost, making the adoption of such models more feasible and appealing in a clinical context. Radiology-GPT outperforms other instruction-tuned models not specially trained for radiology, such as Satbility AI’s Stable LM [ 16 ] and Databrick’s Dolly [ 17 ] . The model is trained on the MIMIC-CXR dataset [ 18 ] , affording it a deep understanding of radiology-specific language and content. Key contributions of our work include: • Development of a dedicated, localized large language model for the field of radiology, addressing the privacy and regulatory challenges in the clinical setting. • Radiology-GPT, leveraging the Alpaca instruction-tuning framework, demonstrates superior performance compared to general instruction-tuned models. • Training of the model on a large, richly annotated radiology dataset, ensuring its proficiency in handling complex radiological language and tasks. • Establishing a precedent for the development of localized foundational models in other medical specialties such as Radiation Oncology and Cardiology, encouraging further advancements in the application of LLMs in various healthcare domains. With these advancements, we believe Radiology-GPT holds immense potential to revolutionize the interpretation of radiology reports and other radiology-associated texts, making it a promising tool for clinicians and researchers alike. 2 Related work 2.1 Large language models In recent years, the field of Natural Language Processing (NLP) has undergone a transformative shift with the advent of large language models (LLMs). These models, such as GPT-3 [ 19 ] , GPT-4 [ 2 ] , and PaLM [ 20 ] , PaLM-2 [ 10 ] , offer enhanced language understanding capabilities, transforming the way language is processed, analyzed, and generated [ 1 , 3 ] . A notable shift in this evolution has been the transition from the pre-training and fine-tuning paradigm of BERT [ 4 ] , GPT [ 21 ] , GPT-2 [ 22 ] and their variants [ 23 , 24 , 5 ] . In contrast, LLMs have introduced impressive few-shot and zero-shot learning capabilities achieved through in-context learning, offering a leap in both the performance and application scope of language models. Alongside the growth of these powerful, commercially developed LLMs, the field has also seen the emergence of open-source alternatives. Models such as LLaMA [ 25 ] and Bloom [ 26 ] have extended the reach of LLMs, fostering democratization and inclusivity by offering the research community accessible and replicable models. Building on open-source LLMs, the research community has begun to explore and develop instruction-tuned language models that can effectively interpret and execute complex instructions within the input text. Exemplifying this trend are models such as Alpaca [ 27 ] , StableLM [ 16 ] , and Dolly [ 17 ] . Alpaca, fine-tuned from Meta’s LLaMA 7B model, has shown that even smaller models can achieve behaviors comparable to larger, closed-source models like OpenAI’s text-davinci-003, while being more cost-effective and easily reproducible. 2.2 Domain-specific language models Domain-Specific Language Models (DSLM) are models that are specifically trained or fine-tuned on text data pertaining to a particular domain or sector [ 28 , 29 ] . By focusing on domain-relevant tasks and language patterns, these models seek to offer superior performance for tasks within their specialized domain. For instance, AgriBERT [ 30 ] , a domain-specific variant of BERT, is trained on an extensive corpus of text data related to agriculture and food sciences. It effectively captures specific jargon and language patterns typical of this sector, making it a valuable tool for tasks such as crop disease classification and food quality assessment. In the field of education, SciEdBERT [ 8 ] exemplifies the potential of DSLMs. SciEdBERT is pre-trained on student response data to middle school chemistry and physics questions. It is designed to understand and evaluate the language and reasoning of students in these domains, enabling accurate evaluation of student responses and providing insights into student understanding and misconceptions. Similarly, in healthcare, ClinicalRadioBERT [ 13 ] has been developed for radiation oncology, training on clinical notes and radiotherapy literature to understand and generate radiation oncology-related text. These examples illustrate the diversity and potential of DSLMs in various sectors, enhancing performance on domain-specific tasks by leveraging tailored language patterns and specific knowledge. While LLMs and DSLMs each have their strengths, there is a growing need for models that leverage the strengths of both, which is particularly pronounced in areas with specialized jargon and extensive bodies of text data, such as radiology. Figure 1 : The overall framework of Radiology-GPT. 3 Methodology Our approach to building Radiology-GPT involves a two-step process: preprocessing of the dataset and fine-tuning the model with instruction following. The pipeline of our method can be seen in Figure 1 . 3.1 Dataset and preprocessing We base our work on the publicly available MIMIC-CXR dataset [ 18 ] , a large, publicly available dataset of chest X-rays (CXRs). MIMIC-CXR contains de-identified medical data from over 60,000 patients who were admitted to the Beth Israel Deaconess Medical Center between 2001 and 2012. We focus on the radiology reports available in this dataset, as they provide rich textual information about patients’ imaging findings and the radiologists’ interpretations. The reports typically contain two sections that correspond to each other: "Findings" and "Impression". The "Findings" section includes detailed observations from the radiology images, whereas the "Impression" section includes the summarized interpretations drawn from those observations. To prepare the data for training, we preprocessed these reports to extract the "Findings" and "Impression" sections from each report and organized them into pairs. The preprocessing involved removing irrelevant sections, standardizing terminologies, and handling missing or incomplete sections. Specifically, we excluded ineligible reports through the following operations: (1) remove reports without finding or impression sections, (2) remove reports whose finding section contained less than 10 words, and (3) remove reports whose impression section contained less than 2 words. We apply the official split published by [ 18 ] and finally obtain 122,014/957/1,606 reports for train/val/test set. In addition, we also preprocessed the OpenI dataset [ 31 ] based on the above exclusion operations to function as an independent external test dataset. It is crucial to validate our model’s performance and generalizability across different data sources. Since the official split is not provided, we follow [ 32 ] to randomly divide the dataset into train/val/test sets by 2400:292:576 (total: 3268 reports). The independent nature of the OpenI dataset allowed us to robustly assess our model’s capabilities and understand its practical applicability. Figure 2 : The instruction-tuning process of Radiology-GPT. Figure 3 : Comparisons of the LLMs based on Understandability, Coherence, Relevance, Conciseness, and Clinical Utility. 3.2 Experimental setting In this study, we trained the current version of Radiology-GPT based on Alpaca-7B. The training was conducted on a server with 4 Nvidia A100 80GB GPUs. We utilized LoRA [ 33 ] to facilitate training. More importantly, we followed the LoRA approach because the small size and portable nature of LoRA weights eases model sharing and deployment. The batch size was set to 128 and the learning rate was fixed at 3e-4. For LoRA, we set lora_r, the rank of the low-rank factorization, to 8, and lora_alpha, the scaling factor for the rank, to 16. This was accompanied by a dropout rate of 0.05. The target modules for LoRA were set to "q_proj" and "v_proj". These modules refer to the query and value matrices in the self-attention mechanism of the transformer architecture [ 34 , 33 ] . A very similar training process will be applied to larger versions of Radiology-GPT based on larger base models. 3.3 Instruction tuning The next step involves instruction tuning our LLM on this radiology dataset. Our methodology is based on the principle of instruction-tuning [ 35 , 27 , 36 ] . The aim is to tune the model such that it can generate an "Impression" text given a "Findings" text as an instruction. The underlying language model learns the relationship between "Findings" and "Impression" from the dataset and hence, starts generating impressions in a similar manner when presented with new findings. Currently, we use Alpaca as the base model for this process. We will release versions of Radiology-GPT in various sizes in the near future. The model also learns domain-specific knowledge relevant to radiology during this training process. To ensure our model learns effectively, we use a specific format for the instructions during training. We provide a short instruction to the "Findings" text: "Derive the impression from findings in the radiology report". The "Impression" text from the same report serves as the target output. This approach promotes learning by aligning the model with the task, thereby creating an instruction-following language model fine-tuned for radiology reports. Examples can be seen in Figure 2 . Through this methodology, Radiology-GPT is designed to capture the specific language patterns, terminologies, and logical reasoning required to interpret radiology reports effectively, thereby making it an efficient and reliable tool for aiding radiologists. It might be valuable to use diverse instruction pairs beyond "Findings —> Impression" in radiology. Currently, "Findings —> Impression" is the most natural and clinically meaningful task to conduct instruction-tuning. We are actively engaging with radiologists to construct a variety of clinically meaningful instruction tuning pairs to further enhance Radiology-GPT. 4 Evaluation One of the challenging aspects of using large language models (LLMs) in the medical field, particularly in radiology, is the determination of their effectiveness and reliability. Given the consequences of errors, it’s crucial to employ appropriate methods to evaluate and compare their outputs. To quantitatively evaluate the effectiveness of Radiology-GPT and other language models in radiology, we implemented a strategy to measure the understandability, coherence, relevance, conciseness, clinical utility of generated responses. It should be noted that even trained radiologists have different writing styles [ 37 , 38 ] . There could be significant variations in how different radiologists interpret the same set of radiology findings [ 39 , 40 , 41 ] . Therefore, we believe it is not appropriate to use string-matching [ 42 ] , BLEU [ 43 ] , ROUGE [ 44 ] , or other n-gram based methods to evaluate the generated radiology impressions. Instead, we develop a set of metrics that are directly relevant to clinical radiology practices. We describe the five metrics below: • Understandability: This metric assesses whether a human reader can understand the content generated by the LLM. Radiologists could rate the understandability of the generated impression section. • Coherence: This metric assesses whether the LLM’s output makes logical sense from beginning to end. For example, in a radiology report, it is necessary to evaluate whether the impression follows logically from the findings. • Relevance: This metric measures whether the impression generated by the LLM is relevant to the context and comprehensively covers core findings. • Conciseness: This metric examines the succinctness of the LLM’s output. The generated content should contain necessary and relevant information, without superfluous details or verbose explanations. • Clinical Utility: This is a key metric for any medical application. It evaluates whether the LLM’s output is useful in a clinical context. Are the impressions generated actually useful for making diagnoses and treatment decisions? Figure 4 : Results from ChatGPT. Figure 5 : LLM comparison. The experimental results yield insightful results regarding the performance of Radiology-GPT in comparison with other existing models. Figure 3 shows the results of the expert evaluation of LLMs. A panel of two expert radiologists assessed the capacity of these LLMs in generating appropriate impressions based on given findings, considering five key parameters: understandability, coherence, relevance, conciseness, and clinical utility. Each metric was scored on a scale from 1 to 5. The radiologists independently assessed the quality of the LLMs’ responses using a set of 10 radiology reports randomly selected from the test set of the MIMIC-CXR radiology reports and the test set of the OpenI dataset preprocessed by our in-house team. The assessments of the two radiologists were conducted independently, and the final scores for each metric were computed by averaging the scores from both radiologists, providing a balanced and comprehensive evaluation of each model’s performance. The performance of Radiology-GPT was found to be comparable to that of ChatGPT in terms of understandability and slightly better in coherence. However, it lagged slightly behind in relevance, not due to a propensity to omit critical findings but rather, because it tended to produce shorter responses compared to ChatGPT. ChatGPT often generated lengthier responses that addressed nearly every point detailed in the findings (which sometimes can provide more context). Figure 4 shows the results of ChatGPT. This discrepancy could be attributed to the contrasting objectives of these models, with Radiology-GPT being designed to deliver succinct and focused outputs that capture the essential aspects of radiological impressions, a critical quality appreciated in the medical field. Consequently, this led to Radiology-GPT scoring higher in both conciseness and clinical utility. In contrast, the other tested models, including StableLM-7B, Dolly-12B, and LLaMA-7B, were all outperformed by both Radiology-GPT and ChatGPT. Figure 5 shows the results of these models. Despite Dolly-12B possessing a larger model size than Radiology-GPT 7B, it could not match the performance of our domain-tuned model. The lack of instruction tuning on domain-specific data and tasks within the field of radiology severely affected the performance of both StableLM-7B and Dolly-12B. LLaMA-7B, not having been instruction-tuned at all, struggled the most. It struggled to comprehend the given instructions and possesses insufficient domain-specific knowledge, leading to a markedly lower performance than other models. These findings underline the significant value that domain-specific tuning and instruction comprehension bring to the capabilities of LLMs in healthcare applications. Figure 6 : An example of deriving diagnostic impression from radiology findings. 5 Discussion The exploration of large language models (LLMs), notably a locally deployable version specialized for radiology—hereafter referred to as Radiology-GPT—in the domains of medicine, and more pointedly in radiology, presents an intriguing trajectory with numerous potential future courses. In this section, we discuss future directions that are relevant to applying LLMs to radiology. Figure 7 : A conversational example. 5.1 Clinical Decision Support Radiology-GPT can provide advice on various tasks, such as determining the best imaging modalities for specific clinical situations or assisting in the preparation of radiology reports. Figure 6 gives a perfect example. It’s important to emphasize that the current generation of LLMs, including Radiology-GPT, ChatGPT, and GPT-4, are designed to augment professional judgment, not replace the roles of medical experts. Future development should focus on amplifying LLMs’ proficiency in deciphering intricate and multifaceted medical terminology, thus amplifying their relevance and reliability in clinical decision-making. The integration of Radiology-GPT with other AI modalities, such as computer vision models proficient in image interpretation, may well represent a significant progression. 5.2 Augmentation of Patient Communication Another potential course is employing Radiology-GPT for the augmentation of patient communication. By transmuting complex medical vernacular and generating intelligible reports, this model could promote a higher level of understanding of medical information among patients. Figure 7 shows a conversational example. Nonetheless, its current capacity to produce precise and trustworthy patient communications without stringent supervision remains restricted. Future endeavours should aim to refine Radiology-GPT to reduce inconsistencies and inaccuracies in its outputs and generate higher quality responses that are grounded in domain knowledge. This might be achievable through the fusion of enhanced domain-specific data and superior training strategies. Concurrently, it is crucial to engineer methods for effective validation and quality control of the generated communications prior to their delivery to patients. 5.3 AGI Expert Panel In a vision of future healthcare scenarios, we envisage the deployment of localized domain-specific LLMs across different medical specialties that would coalesce into a virtual expert panel. Similar to interdisciplinary board panels of physicians who convene to discuss complex cases, such as those in oncology, these AI models could interactively aid in comprehensive decision-making. In the context of oncology, for instance, a patient’s case could be deliberated upon by a Radiology-GPT, a Pathology-GPT, a Oncology-GPT, and a Surgery-GPT, each providing insights from their respective domains. The Radiology-GPT might provide insights about the tumor size, its location, and spread based on the analysis of radiological scans. The Pathology-GPT could provide details on the cancer subtype and the cellular characteristics based on the biopsy report. The Oncology-GPT and Surgery-GPT might provide insights about potential treatment protocols and surgical options. All of these, combined together, could help in the creation of a comprehensive and personalized care plan. While this concept presents significant potential to enhance the thoroughness of clinical decision-making, it also poses several challenges and questions. For instance, the coordination and interaction between these specialized LLMs would require a sophisticated communication framework. Moreover, there are potential risks associated with the aggregation of these insights, especially in cases of conflicting recommendations. Consequently, it would be critical to ensure human oversight to validate and reconcile the conclusions drawn by this AI panel. Future research should focus on developing protocols for AI interaction, determining the most effective methods of integrating their insights, and establishing rigorous checks and balances to ensure the validity and reliability of the collective outputs. Equally significant, the development of such an integrated AI panel system should be guided by robust ethical frameworks to ensure that the autonomy and privacy of the patient are not compromised. Such an AI expert panel system represents a groundbreaking direction in the evolution of LLMs in healthcare, promising to harness the specialized knowledge of various medical fields to provide an unprecedented level of support in complex medical decision-making. In summary, it is crucial to embrace the future of Radiology-GPT in medicine with both a degree of optimism and an attitude of circumspection. Active engagement with large language models, rigorous validation of their outputs, and ethical considerations, particularly relating to patient privacy and data usage, should guide their future evolution and deployment. This approach will ensure that the utilization of Radiology-GPT in healthcare will contribute positively to the progression of patient care, academic research, and the advancement of the medical profession as a whole. 5.4 Privacy and ethics Privacy and ethics are significant considerations when integrating AI technologies in health care. Radiology-GPT, given its nature as a locally trained and locally deployed model, upholds patient privacy. This is an area where local models show their superiority over commercially developed LLMs such as ChatGPT or PaLM-2 [ 10 ] . By operating within the confines of the hosting hospital’s servers, Radiology-GPT mitigates the risk of patient Protected Health Information (PHI) leaks, making the approach compliant with the Health Insurance Portability and Accountability Act (HIPAA) regulations [ 11 ] . However, it is crucial to acknowledge the potential ethical risks that come with the deployment of such models. One of these risks is the possibility of dispensing inaccurate or ungrounded medical advice [ 45 ] . The application of LLMs in a medical setting could inadvertently misinform or misdirect care, especially in the absence of appropriate regulation and control mechanisms. Thus, it is essential to ensure rigorous oversight and regular auditing of these models’ performance. Additionally, bias in training data is a persistent concern in AI, with repercussions potentially amplified in the healthcare context [ 46 ] . If the data used to train Radiology-GPT reflects systematic bias in patient treatment or diagnosis, the model could propagate and even exacerbate these biases. Consequently, fairness, accountability, and transparency in model training are paramount to avoid such pitfalls. It is also necessary to consider the risk of exploitation of these models to generate counterfeit content or misinformation [ 47 ] . The impressive language generation capabilities of LLMs can be manipulated to create misleading or entirely false health information, with potential consequences for public health. 5.5 Expanding Instruction Tuning Data in Radiology In our current work, we utilized "Findings —> Impression" pairs as our primary instruction tuning data. While this forms a crucial part of clinical radiology, the potential utility of Radiology-GPT can be greatly enhanced by constructing and incorporating more diverse and clinically meaningful instruction pairs. In our work, "Findings —> Impression" serves as a natural starting point, mimicking the real-life task of radiologists interpreting findings to provide an impression. However, the field of radiology is rich with a multitude of other tasks and workflows that could potentially be translated into instruction pairs for Radiology-GPT. For instance, instructions could be formulated to generate more patient-friendly explanations of radiology reports or to write up recommendations for further diagnostic tests or treatment based on specific findings. Another potential use case could be to instruct Radiology-GPT to generate a list of differential diagnoses based on presented findings, or to summarize the latest research on specific imaging findings or diseases. Moreover, as radiology practices involve a considerable amount of protocol decision-making [ 48 , 49 ] , Radiology-GPT could also be tuned to suggest optimal imaging protocols based on a given clinical scenario. Expanding the range of instruction pairs can potentially enhance Radiology-GPT’s versatility, making it a more holistic tool that can address different aspects of the radiology workflow. To achieve this, active engagement with practicing radiologists is crucial. By collaborating with experts in the field, we can identify a broad range of real-world tasks that Radiology-GPT can be trained to perform, ultimately ensuring that the model’s capabilities are aligned with the needs of clinical practice. 6 Conclusion In conclusion, we have developed Radiology-GPT, a domain-specific large language model that addresses the critical need for a locally deployable AI solution within the field of radiology. By leveraging the strength of large language models and tailoring it to radiological contexts, Radiology-GPT offers a promising leap forward, displaying superior performance over existing baselines in our evaluations. The evaluation metrics we proposed, which encapsulate a combination of qualitative and quantitative measures, offer a robust framework for assessing the effectiveness of this and similar models within healthcare settings. Moreover, Radiology-GPT opens up exciting avenues for future applications. It provides a foundation for the incorporation of multimodal data, including radiological images, further enhancing its potential contributions to the field. Its localized nature also paves the way for wider applications in other medical specialties, stimulating advancements in healthcare AI that respect privacy regulations. Our study stands as a testament to the potential of specialized AI in medicine, offering both immediate benefits and laying the groundwork for future innovation. References [1] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852 , 2023. [2] R OpenAI. Gpt-4 technical report. arXiv , 2023. [3] Lin Zhao, Lu Zhang, Zihao Wu, Yuzhong Chen, Haixing Dai, Xiaowei Yu, Zhengliang Liu, Tuo Zhang, Xintao Hu, Xi Jiang, et al. When brain-inspired ai meets agi. arXiv preprint arXiv:2303.15935 , 2023. [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. [5] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419 , 2023. [6] Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, et al. Chataug: Leveraging chatgpt for text data augmentation. arXiv preprint arXiv:2302.13007 , 2023. [7] Chong Ma, Zihao Wu, Jiaqi Wang, Shaochen Xu, Yaonai Wei, Zhengliang Liu, Lei Guo, Xiaoyan Cai, Shu Zhang, Tuo Zhang, et al. Impressiongpt: An iterative optimizing framework for radiology report summarization with chatgpt. arXiv preprint arXiv:2304.08448 , 2023. [8] Zhengliang Liu, Xinyu He, Lei Liu, Tianming Liu, and Xiaoming Zhai. Context matters: A strategy to pre-train language model for science education. arXiv preprint arXiv:2301.12031 , 2023. [9] Zihao Wu, Lu Zhang, Chao Cao, Xiaowei Yu, Haixing Dai, Chong Ma, Zhengliang Liu, Lin Zhao, Gang Li, Wei Liu, et al. Exploring the trade-offs: Unified large language models vs local fine-tuned models for highly-specific radiology nli task. arXiv preprint arXiv:2304.09138 , 2023. [10] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023. [11] Zhengliang Liu, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Wei Liu, Dinggang Shen, Quanzheng Li, et al. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032 , 2023. [12] An Yan, Julian McAuley, Xing Lu, Jiang Du, Eric Y Chang, Amilcare Gentili, and Chun-Nan Hsu. Radbert: Adapting transformer-based language models to radiology. Radiology: Artificial Intelligence , 4(4):e210258, 2022. [13] Saed Rezayi, Haixing Dai, Zhengliang Liu, Zihao Wu, Akarsh Hebbar, Andrew H Burns, Lin Zhao, Dajiang Zhu, Quanzheng Li, Wei Liu, et al. Clinicalradiobert: Knowledge-infused few shot learning for clinical notes named entity recognition. In Machine Learning in Medical Imaging: 13th International Workshop, MLMI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings , pages 269–278. Springer, 2022. [14] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022. [15] Tianyang Zhong, Yaonai Wei, Li Yang, Zihao Wu, Zhengliang Liu, Xiaozheng Wei, Wenjun Li, Junjie Yao, Chong Ma, Xiang Li, et al. Chatabl: Abductive learning via natural language interaction with chatgpt. arXiv preprint arXiv:2304.11107 , 2023. [16] Anel Islamovic. Stability AI Launches the First of its StableLM Suite of Language Models — Stability AI — stability.ai. https://stability.ai/blog/stability-ai-launches-the-first-of-its-stablelm-suite-of-language-models . [Accessed 09-Jun-2023]. [17] Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM — databricks.com. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm . [Accessed 09-Jun-2023]. [18] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data , 6(1):317, 2019. [19] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020. [20] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. [21] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. [22] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. [23] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019. [24] Wenxiong Liao, Zhengliang Liu, Haixing Dai, Zihao Wu, Yiyang Zhang, Xiaoke Huang, Yuzhong Chen, Xi Jiang, Dajiang Zhu, Tianming Liu, et al. Mask-guided bert for few shot text classification. arXiv preprint arXiv:2302.10447 , 2023. [25] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. [26] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022. [27] Stanford CRFM — crfm.stanford.edu. https://crfm.stanford.edu/2023/03/13/alpaca.html . [Accessed 09-Jun-2023]. [28] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) , 3(1):1–23, 2021. [29] Zhengliang Liu, Mengshen He, Zuowei Jiang, Zihao Wu, Haixing Dai, Lian Zhang, Siyi Luo, Tianle Han, Xiang Li, Xi Jiang, et al. Survey on natural language processing in medical image analysis. Zhong nan da xue xue bao. Yi xue ban= Journal of Central South University. Medical Sciences , 47(8):981–993, 2022. [30] Saed Rezayi, Zhengliang Liu, Zihao Wu, Chandra Dhakal, Bao Ge, Chen Zhen, Tianming Liu, and Sheng Li. Agribert: knowledge-infused agricultural language models for matching food and nutrition. IJCAI, 2022. [31] Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association , 23(2):304–310, 2016. [32] Jinpeng Hu, Jianling Li, Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan, and Tsung-Hui Chang. Word graph guided summarization for radiology findings. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 4980–4990, 2021. [33] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. [35] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. [36] Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. arXiv preprint arXiv:2305.08809 , 2023. [37] A Wallis and P McCoubrie. The radiology report—are we getting the message across? Clinical radiology , 66(11):1015–1022, 2011. [38] Ricky K Taira, Stephen G Soderland, and Rex M Jakobovits. Automatic structuring of radiology free-text reports. Radiographics , 21(1):237–245, 2001. [39] Joann G Elmore, Carolyn K Wells, Carol H Lee, Debra H Howard, and Alvan R Feinstein. Variability in radiologists’ interpretations of mammograms. New England Journal of Medicine , 331(22):1493–1499, 1994. [40] David Gur, Andriy I Bandos, Cathy S Cohen, Christiane M Hakim, Lara A Hardesty, Marie A Ganott, Ronald L Perrin, William R Poller, Ratan Shah, Jules H Sumkin, et al. The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology , 249(1):47–53, 2008. [41] Geoffrey A Sonn, Richard E Fan, Pejman Ghanouni, Nancy N Wang, James D Brooks, Andreas M Loening, Bruce L Daniel, Katherine J To’o, Alan E Thong, and John T Leppert. Prostate magnetic resonance imaging interpretation varies substantially across radiologists. European urology focus , 5(4):592–599, 2019. [42] KM Alhendawi and Ahmad Suhaimi Baharudin. String matching algorithms (smas): survey & empirical analysis. Journal of Computer Sciences and Management , 2013. [43] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318, 2002. [44] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. [45] Malik Sallam. The utility of chatgpt as an example of large language models in healthcare education, research and practice: Systematic review on the future perspectives and potential limitations. medRxiv , pages 2023–02, 2023. [46] Christopher J Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. Key challenges for delivering clinical impact with artificial intelligence. BMC medicine , 17:1–9, 2019. [47] Wenxiong Liao, Zhengliang Liu, Haixing Dai, Shaochen Xu, Zihao Wu, Yiyang Zhang, Xiaoke Huang, Dajiang Zhu, Hongmin Cai, Tianming Liu, et al. Differentiate chatgpt-generated and human-written medical texts. arXiv preprint arXiv:2304.11567 , 2023. [48] Daniel I Glazer, Pamela J DiPiro, Atul B Shinagare, Raymond Y Huang, Aijia Wang, Giles W Boland, and Ramin Khorasani. Ct and mri protocol variation and optimization at an academic medical center. Journal of the American College of Radiology , 15(9):1254–1258, 2018. [49] Andrew J Einstein, Daniel S Berman, James K Min, Robert C Hendel, Thomas C Gerber, J Jeffrey Carr, Manuel D Cerqueira, S James Cullom, Robert DeKemp, Neal W Dickert, et al. Patient-centered imaging: shared decision making for cardiac imaging procedures with exposure to ionizing radiation. Journal of the American College of Cardiology , 63(15):1480–1489, 2014.
5. Training Protocol, Convergence Behavior & Experimental Evaluation
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Model Figures & Evaluation Curves (4 Images)
Deep Learning Execution Pipeline
(Preprocessing)
(Feature Extraction)
(Self-Attention)
(Loss Optimization)