Peft Parameter Efficient Fine Tuning | AI Final Year Project Report

Complete Technical Paper Content — Peft Parameter Efficient Fine Tuning

1. Executive Abstract, Problem Formulation & Background

In deep learning, fine-tuning is the process of adapting a computational model trained for one task (the upstream task) to perform a different, usually more specific, task (the downstream task). It is considered a form of transfer learning, as it reuses knowledge learned from the original training objective. Fine-tuning involves applying additional training (e.g., on new data) to the parameters of a neural network that have been pre-trained. Many variants exist. The additional training can be applied to the entire neural network, or to only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen" (i.e., not changed during backpropagation). A model may also be augmented with "adapters"—lightweight modules inserted into the model's architecture that nudge the embedding space for domain adaptation. These contain far fewer parameters than the original model and can be fine-tuned in a parameter-efficient way by tuning only their weights and leaving the rest of the model's weights frozen. For some architectures, such as convolutional neural networks, it is common to keep the earlier layers (those closest to the input layer) frozen, as they capture lower-level features, while later layers often discern high-level features that can be more related to the task that the model is trained on. Models that are pre-trained on large, general corpora are usually fine-tuned by reusing their parameters as a starting point and adding a task-specific layer trained from scratch. Fine-tuning the full model is also common and often yields better results, but is more computationally expensive. Fine-tuning is typically accomplished via supervised learning, but there are also techniques to fine-tune a model using weak supervision. Fine-tuning can be combined with a reinforcement learning from human feedback-based objective to produce language models such as ChatGPT (a fine-tuned version of GPT models) and Sparrow.

Fine-tuning large pretrained models is often prohibitively costly due to their scale. Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of large pretrained models to various downstream applications by only fine-tuning a small number of (extra) model parameters instead of all the model's parameters. This significantly decreases the computational and storage costs. Recent state-of-the-art PEFT techniques achieve performance comparable to fully fine-tuned models.

PEFT is integrated with Transformers for easy model training and inference, Diffusers for conveniently managing different adapters, and Accelerate for distributed training and inference for really big models.

Visit the PEFT organization to read about the PEFT methods implemented in the library and to see notebooks demonstrating how to apply these methods to a variety of downstream tasks. Click the "Watch repos" button on the organization page to be notified of newly implemented methods and notebooks!

Check the PEFT Adapters API Reference section for a list of supported PEFT methods, and read the Adapters , Soft prompts , and IA3 conceptual guides to learn more about how these methods work.

2. Related Work, Literature Survey & Algorithmic Foundations

Prepare a model for training with a PEFT method such as LoRA by wrapping the base model and PEFT configuration with get_peft_model . For the bigscience/mt0-large model, you're only training 0.19% of the parameters!

import torch from transformers import AutoModelForCausalLM from peft import LoraConfig , TaskType , get_peft_model device = torch . accelerator . current_accelerator (). type if hasattr ( torch , "accelerator" ) else "cuda" model_id = "Qwen/Qwen2.5-3B-Instruct" model = AutoModelForCausalLM . from_pretrained ( model_id , device_map = device ) peft_config = LoraConfig ( r = 16 , lora_alpha = 32 , task_type = TaskType . CAUSAL_LM , # target_modules=["q_proj", "v_proj", ...] # optionally indicate target modules ) model = get_peft_model ( model , peft_config ) model . print_trainable_parameters () # prints: trainable params: 3,686,400 || all params: 3,089,625,088 || trainable%: 0.1193 # now perform training on your dataset, e.g. using transformers Trainer, then save the model model . save_pretrained ( "qwen2.5-3b-lora" )

There are many benefits of using PEFT but the main one is the huge savings in compute and storage, making PEFT applicable to many different use cases.

Consider the memory requirements for training the following models on the ought/raft/twitter_complaints dataset with an A100 80GB GPU with more than 64GB of CPU RAM.

With LoRA you can fully finetune a 12B parameter model that would've otherwise run out of memory on the 80GB GPU, and comfortably fit and train a 3B parameter model. When you look at the 3B parameter model's performance, it is comparable to a fully finetuned model at a fraction of the GPU memory.

3. Dataset Ingestion, Preprocessing & Stochastic Augmentations

The bigscience/T0_3B model performance isn't optimized in the table above. You can squeeze even more performance out of it by playing around with the input instruction templates, LoRA hyperparameters, and other training related hyperparameters. The final checkpoint size of this model is just 19MB compared to 11GB of the full bigscience/T0_3B model. Learn more about the advantages of finetuning with PEFT in this blog post .

Quantization is another method for reducing the memory requirements of a model by representing the data in a lower precision. It can be combined with PEFT methods to make it even easier to train and load LLMs for inference.

PEFT can help you save storage by avoiding full finetuning of models on each of downstream task or dataset. In many cases, you're only finetuning a very small fraction of a model's parameters and each checkpoint is only a few MBs in size (instead of GBs). These smaller PEFT adapters demonstrate performance comparable to a fully finetuned model. If you have many datasets, you can save a lot of storage with a PEFT model and not have to worry about catastrophic forgetting or overfitting the backbone or base model.

PEFT is widely supported across the Hugging Face ecosystem because of the massive efficiency it brings to training and inference.

The iterative diffusion process consumes a lot of memory which can make it difficult to train. PEFT can help reduce the memory requirements and reduce the storage size of the final model checkpoint. For example, consider the memory required for training a Stable Diffusion model with LoRA on an A100 80GB GPU with more than 64GB of CPU RAM. The final model checkpoint size is only 8.8MB!

4. Neural Network Backbone, Layer Specifications & Loss Formulations

Take a look at the examples/lora_dreambooth/train_dreambooth.py training script to try training your own Stable Diffusion model with LoRA, and play around with the smangrul/peft-lora-sd-dreambooth Space which is running on a T4 instance. Learn more about the PEFT integration in Diffusers in this tutorial .

PEFT is directly integrated with Transformers . After loading a model, call add_adapter to add a new PEFT adapter to the model:

from peft import LoraConfig model = ... # transformers model peft_config = LoraConfig (...) model . add_adapter ( peft_config , adapter_name = "lora_1" )

model = ... # transformers model model . load_adapter ( < path - to - adapter > , adapter_name = "lora_1" )

The Transformers integration doesn't include all the functionalities offered in PEFT, such as methods for merging the adapter into the base model.

5. Training Protocol, Convergence Behavior & Experimental Evaluation

Accelerate is a library for distributed training and inference on various training setups and hardware (GPUs, TPUs, Apple Silicon, etc.). PEFT models work with Accelerate out of the box, making it really convenient to train really large models or use them for inference on consumer hardware with limited resources.

PEFT can also be applied to training LLMs with RLHF components such as the ranker and policy. Get started by reading:

Use this Space or check out the docs to find which models officially support a PEFT method out of the box. Even if you don't see a model listed below, you can manually configure the model config to enable PEFT for a model. Read the New transformers architecture guide to learn how.

To use 🤗 PEFT in your publication, please cite it by using the following BibTeX entry.

@Misc { peft , title = { {PEFT}: State-of-the-art Parameter-Efficient Fine-Tuning methods } , author = { Sourab Mangrulkar and Sylvain Gugger and Lysandre Debut and Younes Belkada and Sayak Paul and Benjamin Bossan and Marian Tietz } , howpublished = { \url{https://github.com/huggingface/peft} } , year = { 2022 } }