Fine-Tuning Llama 3 and Deploying It for Inference
Hassaan Qadir
Let’s say you want to fine-tune llama3. By running PEFT LoRA fine-tuning on a dataset with a few thousand rows in it, you can dramatically increase the accuracy of your model’s responses, and if you use Beam for compute, you can accomplish it in 1 hour rather than 10.
Setup Beam and Llama3 model
1. Set up the Beam API token
First, install the Beam client, which will ask for an API key.
Go to beam.cloud and create a Pay-As-You-Go Developer account. You’ll receive the API key and instructions on how to save it to your environment. Follow these instructions.
2. Access Llama3 weights
Next, you need the weights of the Llama3 base model to begin fine-tuning them. Go here to request access, which is typically granted immediately, along with a custom URL and instructions on using it to download the weights. Follow the instructions.
3. Upload the weights and dataset to a Beam Volume
Finally, upload the weights and your fine-tuning dataset to a Beam Volume.
To create a fine-tuning dataset, you need a collection of input-output pairs, formatted as a CSV, JSON, or text file, where each entry provides an example of what the model should predict given a specific input. Reference these examples from Meta for Llama3 formatting.
Once you have your fine-tuning dataset, use the following CLI commands.
Fine-tune using LoRA
1. Configure LoRA
We will be leveraging the transfomers and peft packages to run LoRA PEFT fine-tuning, with the same Python as you would for a local run. Here is the default configuration we will be working with.
2. Configure Beam
That’s all set to run locally, but it would take hours and hours on consumer hardware. Fortunately, we can run it on Beam’s serverless GPUs by adding a few lines of code.
Within the @function decorator, we’ve done three things. First, we set paths to the Volume we created earlier. Second, we mounted an Image, which gives Beam access to the Python packages we describe. And third, we selected the GPU and CPU specs for our run.
Outside of the decorator, we modified the script to call llama_fine_tune() with the remote() method, which will send the compute to the Beam CLI.
3. Run the fine-tuning function through the Beam CLI
The moment of truth! Simply run
in the terminal, and you should receive output like this:
Once this is complete, use the Beam CLI to check on the output files:
Using your Fine-Tuned Model
1. Write an inference function
Start off by writing a local function to call inference on the model, customizing it to the format of your fine-tuning dataset:
2. Add the endpoint decorator and a function to load the model
Like before, we’ll add a decorator to this function to allow Beam to access it, but we’ll also write another function, load_finetuned_model() that we pass into the decorator so Beam knows how to load the model:
3. Deploy the endpoint and make API calls
Finally, it’s time to deploy our endpoint! Last time, we were running a one-off function, so it was enough to call the script using:
But we want to set up a persistent endpoint, so we'll run the following command:
This should return the following output:
Congrats! Your very own fine-tuned model is now running and accessible through POST requests like:
which could return:
Useful Links
https://huggingface.co/docs/transformers/en/training
https://llama.meta.com/docs/how-to-guides/fine-tuning/
https://github.com/meta-llama/llama-recipes/tree/main/recipes/quickstart/finetuning/datasets
