Deploy a Fine-Tuned LLM as a Dedicated Endpoint

To deploy a fine-tuned LLM as a dedicated endpoint, merge the LoRA adapter into the base weights, store the merged model on a volume, and serve it with vLLM on a GPU behind an authenticated URL. On Beam that is one VLLM object pointed at the volume and beam deploy. The endpoint speaks the OpenAI API, runs only your model, and scales to zero between requests.

Deploy it with beam deploy app.py:support_llm, add /v1 to the URL it prints, and call it with the OpenAI SDK:

The example assumes a LoRA fine-tune of Llama 3.1 8B Instruct. Merged in 16-bit precision, the weights take about 16 GB, which fits on the RTX 4090's 24 GB with a few gigabytes left for the KV cache.

What deploying a fine-tuned LLM involves

Deploying a fine-tuned LLM means turning a training output (usually a LoRA adapter folder or a full checkpoint) into a private API that other code can call. Training gets most of the attention in fine-tuning guides, including our own Llama 3 fine-tuning walkthrough. Serving is a separate problem with its own failure modes.

You need:

• Servable weights. A merged safetensors checkpoint with its tokenizer, or a base model plus an adapter the server knows how to load.
• A GPU sized to the model. Weights plus KV cache for your context length must fit in GPU memory.
• An inference server. vLLM is the common choice because it batches concurrent requests and exposes OpenAI-compatible routes.
• A private URL with TLS and auth. A fine-tuned model often encodes your data, so it shouldn't sit on an open port.
• Scaling and idle cost. Traffic to an internal model is usually bursty, and a GPU billed around the clock is the largest cost for low-traffic endpoints.

"Dedicated" here means the endpoint runs your weights in containers that serve only you, unlike a shared per-token API where many customers' requests run on the same model.

Merge the LoRA adapter or serve it separately?

Merge the adapter when you have one fine-tune per base model. Merging folds the low-rank update into the base weights, so the server loads one ordinary checkpoint and inference has no per-request adapter overhead. Serve adapters separately when you have many fine-tunes of the same base and want them to share one GPU.

This guide takes the merged path, which covers most teams shipping one production fine-tune. The multi-adapter setup is covered in the FAQ.

How to deploy a fine-tuned LLM on a serverless GPU

The flow has four steps: upload the adapter to a volume, merge it into the base model on Beam, serve the merged checkpoint with vLLM, and call it with any OpenAI client. Everything stays on one volume, so the weights never leave your workspace and later cold starts read them from storage instead of downloading them again.

Upload the adapter to a volume

Install the CLI, create a volume, and copy your adapter folder into it. The folder is whatever your trainer saved: adapter_config.json, adapter_model.safetensors, and usually the tokenizer files.

If you trained on Beam, as in our Unsloth fine-tuning guide, the adapter is already on a volume and you can skip the upload.

Merge the adapter into the base weights

Run the merge as a Beam function so the base model downloads next to the volume rather than to your laptop. Llama 3.1 is a gated model, so store your Hugging Face token as a secret first with beam secret create HF_TOKEN <your-token>.

Run it with python merge.py. AutoPeftModelForCausalLM reads the base model name from adapter_config.json, downloads it, and applies the adapter. The merge itself is quick. Most of the time goes to downloading and loading 16 GB of weights, which is why this runs on CPU with plenty of RAM instead of a GPU.

If your adapter folder has no tokenizer files, load the tokenizer from the base model's Hugging Face ID instead. Ship the tokenizer you trained with: a fine-tune that added special tokens or a custom chat template breaks quietly when served with the base tokenizer.

Serve the merged model with vLLM

The hero snippet mounts the same finetunes volume and points VLLMArgs.model at the merged folder. vLLM accepts a local directory anywhere it accepts a Hugging Face ID, so no extra download step is needed. served_model_name sets the name clients pass as model, which keeps the internal path out of your API.

Two settings matter for throughput. concurrent_requests=32 lets vLLM batch up to 32 requests on each container; the class default of 1 sends one request at a time and wastes most of the GPU. tasks_per_container on the autoscaler should match it, so a second container starts only when the first is full.

Save the snippet as app.py and run beam deploy app.py:support_llm. The endpoint requires your Beam token as a Bearer token by default, which is exactly what the OpenAI SDK sends as api_key.

Call it with the OpenAI SDK

Any client that speaks the OpenAI API works after you change the base URL, key, and model name. That includes LangChain, LlamaIndex, and the official SDKs. You can also set OPENAI_BASE_URL and OPENAI_API_KEY and leave existing code unchanged. The OpenAI-compatible API guide covers streaming, tool calling, and the other routes the server exposes.

Problems that break fine-tuned model serving

Most failed deployments of a fine-tuned model come from mismatches between how the model was trained and how it's served, not from the infrastructure. Check these before you debug the GPU.

• Missing chat template. A fine-tune that started from a base model, not an Instruct model, often ships a tokenizer with no chat template, and /v1/chat/completions fails. Pass the template your training data used through chat_template in VLLMArgs, or call /v1/completions with the raw prompt format.
• Merging into a 4-bit base. QLoRA and Unsloth's bnb-4bit checkpoints train against quantized weights. Merge the adapter into the 16-bit base model instead, or use Unsloth's save_pretrained_merged(..., save_method="merged_16bit"). Merging into the quantized weights loses quality.
• Adapter rank above the server limit. vLLM's LoRA mode rejects adapters with a rank above max_lora_rank, which defaults to 16. Merged models don't have this limit.
• Prompt format drift. If you trained on an Alpaca-style template and serve through a Llama 3 chat template, the model sees inputs it never learned from. Answers get worse without any error.
• Context length versus memory. Each 8,192-token sequence on Llama 3.1 8B needs about 1 GB of KV cache in 16-bit precision. Raise max_model_len only as far as your traffic needs, or vLLM will refuse to start, or fit only a few requests at once.

How much a dedicated fine-tuned LLM endpoint costs

On Beam's serverless pricing, the hero configuration costs about $1.76 an hour while a container runs: $0.69 for the RTX 4090, plus 2 CPU cores and 16 GiB of RAM billed at the GPU-attached rates. It costs nothing while the endpoint has scaled to zero. Billing is by the millisecond, only while your code runs.

A support assistant busy 8 hours a day on weekdays runs about 176 hours a month, or about $310. The same model on an always-on RTX 4090 machine at $0.44 an hour costs about $320 a month. Serverless is cheaper below roughly 25% utilization, and a dedicated machine wins once the endpoint is busy most of the day. keep_warm_seconds=300 keeps a container up for five minutes after the last request, so bursts don't each pay a cold start. Our post on serverless GPU cold starts covers how to shorten the start itself.

Choosing where to host a fine-tuned LLM

The right host depends on how much traffic the model gets, how big it is, and whether you want to own the serving code. Managed fine-tuning platforms make deployment one click but bill dedicated replicas by the hour on large GPUs. Serverless GPU platforms cost less for bursty traffic and leave the server configuration to you.

Rates come from each provider's published pricing page. Modal and Beam also bill CPU and memory on top of the GPU.

Where each one fits:

• Beam suits teams that want the endpoint defined in Python next to their training code, weights kept on their own volume, and a 24 GB serverless GPU that scales to zero. Models that need 80 GB cards run on our dedicated machines instead of serverless.
• Together AI is the shortest path if you fine-tuned there: deploy from the model page or tg endpoints deploy. Replicas bill per minute even when idle, and not every base model can run on a dedicated endpoint.
• Fireworks serves trained models through on-demand deployments and supports multi-LoRA. It starts at H100 class, which is a lot of GPU for an 8B model.
• Hugging Face Inference Endpoints fits teams already pushing models to the Hub. Our Hugging Face Inference Endpoints alternatives guide compares it with other hosts.
• RunPod Serverless has a browser-configured vLLM worker and the same $0.69 rate for its 24 GB class, which covers L4, A5000, and 3090 cards. Its RTX 4090 tier costs $1.10 an hour.
• Modal gives full control over the vLLM process and a wide GPU range. You write the launch code, and the Starter plan includes $30 a month in credits.

For more hosts that serve custom models, see our roundup of Replicate alternatives. The background on our vLLM integration is in serving vLLM on Beam.

FAQ

Should I merge my LoRA adapter before deploying?

Merge it if you serve one fine-tune per base model. A merged checkpoint loads like any other model, has no per-request adapter overhead, and avoids LoRA limits such as maximum rank. Keep adapters separate only when several fine-tunes share a base and you want them on one GPU.

Can one endpoint serve several LoRA adapters?

Yes. vLLM loads a base model with enable_lora, registers each adapter under its own name, and lets clients pick one through the model field. max_loras sets how many adapters stay on the GPU at once, and max_lora_rank must be at least your largest adapter's rank. Requests for different adapters run in the same batch.

How do I deploy a fine-tuned model from a private Hugging Face repo?

Store a Hugging Face token with read access as a secret named HF_TOKEN, pass secrets=["HF_TOKEN"] to the VLLM class, and set model to the repo ID. The container logs in with the token before vLLM downloads the weights, and the default vllm_cache volume keeps them for later cold starts.

How much does it cost to host a fine-tuned LLM?

An 8B model on a serverless RTX 4090 with 2 CPU cores and 16 GiB of RAM costs about $1.76 an hour while it runs on Beam, and nothing while scaled to zero. At 8 busy hours each weekday that is about $310 a month. Managed dedicated endpoints on H100s start at $5.49 an hour per replica.

Does a fine-tuned 8B model need an H100?

No. Llama 3.1 8B or Qwen2.5 7B in 16-bit precision takes 15 to 16 GB, which fits on a 24 GB RTX 4090 with room for KV cache. Larger fine-tunes, such as a 70B model, need 80 GB-class GPUs or several GPUs, and those run on dedicated machines.

How do I avoid cold starts on a fine-tuned model endpoint?

Raise keep_warm_seconds so containers stay up between bursts, or set min_containers=1 on the autoscaler to keep one running at all times. Both bill while idle. Loading weights from a volume, as in this guide, shortens each cold start but doesn't remove the time to load the model onto the GPU.

Ship your fine-tuned model as a private API

Merge the adapter, point the VLLM class at your volume, and deploy. You get an authenticated OpenAI-compatible endpoint that runs only your model and scales to zero when nobody calls it. Deploy your fine-tuned LLM on Beam.