LoRA hyperparameters and what each one costs on your model
Two LoRA settings change how much VRAM a run needs. Rank and the target module set fix the trainable parameter count, and every memory term follows from that count. Alpha, dropout, the learning rate and the schedule change the model you get, not the memory you need. The planner turns your choice into an exact trainable parameter count and a fit verdict on a real card.
The planner
Trainable parameters 167,772,160, which is 2.09 percent of the base model. Adapter file at bf16 320.00 MiB. Effective batch 8 sequences. Shape used, 32 layers, hidden 4096, heads 32, kv heads 8, head_dim 128 (config.json).
| Term, GiB | LoRA, 16 bit base | QLoRA, 4 bit base | Full fine tuning |
|---|---|---|---|
| Frozen base weights | 14.96 | 3.86 | 0.00 |
| Trainable weights, fp16, 2 bytes each | 0.31 | 0.31 | 14.96 |
| Gradients, fp16, 2 bytes each | 0.31 | 0.31 | 14.96 |
| fp32 master copy, 4 bytes each | 0.63 | 0.63 | 29.92 |
| Adam moments m and v, 8 bytes each | 1.25 | 1.25 | 59.83 |
| Activations, estimate | 1.00 | 1.00 | 1.00 |
| Total | 18.46 | 7.36 | 120.66 |
| On GeForce RTX 4090, 24 GB | fits, 5.54 GiB spare | fits, 16.64 GiB spare | short by 96.66 GiB |
LoraConfig( r=64, lora_alpha=16, lora_dropout=0.1, bias="none", target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"], task_type="CAUSAL_LM", )
Constants used, stated rather than buried. The frozen base is counted at 2 bytes per parameter in the 16 bit column and 4 bits per parameter in the quantized column, since QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into low rank adapters (QLoRA abstract). Trainable parameters are counted the ZeRO way, an fp16 copy of the parameters and of the gradients at 2 bytes per parameter each plus Adam state made of an fp32 copy of the parameters, the momentum and the variance at 4 bytes per parameter each (ZeRO section 3.1). The momentum and variance are the 8 bytes per parameter the Transformers memory anatomy page states for Adam (memory anatomy). Activation memory is an estimate, described below. The sequence length box starts at a placeholder value, so replace it with the length you actually train at. Card capacities are nameplate figures treated as GiB, and usable VRAM sits below nameplate because the CUDA context and allocator fragmentation take a share. That share is an estimate, so measure it on your own machine and put it in the reserve field.
Starting values you can defend
Start here. Every value below is one the QLoRA authors used to fine tune large language models, not a preference. Their experiments set LoRA r to 64 and alpha to 16 and add LoRA modules on all linear layers of the base model, and they report that the most critical LoRA hyperparameter is how many LoRA adapters are used in total. Covering every linear layer is the part that matters most, so do that first, then set a rank of 64 and an alpha of 16. They use a LoRA dropout of 0.1 for models up to 13B and 0.05 for 33B and 65B models. Table 9 of that paper lists the per model size training hyperparameters, giving batch size, learning rate, steps, source length and target length, and its rates are 2e-4 at a batch size of 16 for the 7B and 13B models and 1e-4 above that. After benchmarking linear and cosine schedules those authors use a constant learning rate schedule.
Do not carry over the Trainer default learning rate of 5e-05. It is a full fine tuning default and it is low for adapter training. The same paper finds that other LoRA hyperparameters, such as the projection dimension r, do not affect performance once the adapter covers every linear layer, so treat rank as the memory dial rather than the quality dial and use the planner below to see exactly what each rank costs on your model. PEFT itself defaults r to 8 and lora_alpha to 8, which is far smaller than the configuration above.
The table gives every knob, its starting value, and where that value comes from. The kind column says whether it is a library default or a value a published paper actually used, so you can check any of them yourself.
| Setting | Start | Kind | Source and tradeoff |
|---|---|---|---|
| r | 8 | Library default | PEFT defaults r to 8 and describes it as the Lora attention dimension (PEFT reference). Sets the trainable count, so it moves memory. QLoRA searched ranks of 8, 16, 32, 64, 128 and 256 and reports rank unrelated to final performance when LoRA is used on all layers (QLoRA). |
| lora_alpha | 8 | Library default | PEFT defaults it to 8 (PEFT reference). LoRA scales the update by alpha divided by r, and with Adam tuning alpha is roughly the same as tuning the learning rate if the initialisation is scaled appropriately (LoRA). No memory cost. |
| lora_dropout | 0.0 | Library default | PEFT defaults it to 0.0 and defines it as the dropout probability for the Lora layers (PEFT reference). QLoRA used 0.1 for models up to 13B and 0.05 for 33B and 65B (QLoRA). Does not change the parameter count. |
| target_modules | Attention blocks, or all linear layers | Convention and paper value | target_modules names the modules the LoRA update matrices are applied to, and in Transformer models LoRA is typically applied to attention blocks only (PEFT guide). QLoRA found the most critical LoRA hyperparameter is how many adapters are used in total, with all linear layers required to match full finetuning (QLoRA). The second setting that moves memory. |
| bias | none | Library default | PEFT defaults bias to none and accepts none, all or lora_only (PEFT reference). |
| learning_rate | 2e-4 for 7B and 13B, 1e-4 for 33B and up | Paper value | Table 9 of the QLoRA paper lists the per model size training hyperparameters, giving batch size, learning rate, steps, source length and target length, and those are its rates, at a batch size of 16. Adapter training takes a higher rate than full fine tuning, so the Trainer default of 5e-05 is low here. Costs no memory. |
| num_train_epochs | 3.0 | Library default | Trainer default (Trainer reference). Wall clock only. |
| lr_scheduler_type | linear | Library default | Trainer default (Trainer reference). After benchmarking linear and cosine, QLoRA used a constant schedule (QLoRA). |
| per_device_train_batch_size | 8 | Library default | Trainer default (Trainer reference). Moves the activation term, which is usually what decides the fit. |
| gradient_accumulation_steps | 1 | Library default | Trainer default, and it accumulates gradients over update steps before a backward and update pass, simulating a larger batch without additional memory (Trainer reference). Raise this, not the microbatch, when VRAM is tight. |
| gradient_checkpointing | False | Library default | Trainer default. Turning it on trades compute for memory by clearing activations in the forward pass and recomputing them in the backward pass, at roughly 20 percent slower training (Trainer reference). |
| 4 bit base | load_in_4bit=True, nf4, double quant, bfloat16 compute | Docs and paper | Those are the 4 bit settings in the PEFT quantization guide (PEFT quantization), and QLoRA used NF4 with double quantization and bf16 as the computation datatype (QLoRA). After loading, prepare_model_for_kbit_training preprocesses the model for training (PEFT quantization). |
| The QLoRA pairing | r=64, lora_alpha=16, all linear layers | Paper value | The QLoRA experiments set r to 64 and alpha to 16 and add LoRA modules on all linear layers of the base model (QLoRA). Alpha smaller than r, which is the opposite of the common advice. |
| use_rslora | Off | Library option | Setting it True selects Rank-Stabilized LoRA, which sets the scaling factor to lora_alpha divided by the square root of r (PEFT guide). Worth knowing before you sweep rank. |
What a rank of one costs on each model
For one adapted linear layer of shape (out, in), LoRA injects an A of shape (r, in) and a B of shape (out, r), so it adds r times the quantity in plus out. The number of trainable parameters is determined by the rank and the shape of the original weights (LoRA), and PEFT states the same dependence on the rank r and the shape of the original weight matrix (PEFT guide). Because the count is linear in r, one coefficient per model settles every rank.
Here is the correction. Under grouped query attention, query heads are divided into groups, each of which shares a single key head and value head (GQA), so k_proj and v_proj are narrower than q_proj by the query heads per KV head ratio. The popular q and v only recipe therefore costs a different amount on every model in this table. The same ratio corrects a second thing. A KV cache calculator that sizes the cache with num_attention_heads instead of num_key_value_heads over reports it by the percentage in the overstatement column. Every shape below is read from that model's own config.json, linked on the model name, so you can open the file and see the key.
| Model | layers | hidden_size | intermediate_size | heads | kv heads | head_dim | Parameters | q k v o per rank | gate up down per rank | Q heads per KV head | KV overstated by | KV bytes per token | KiB per token | KV GiB at 128k |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 4096 | 14336 | 32 | 8 | 128 derived | 8030261248 | 851968 | 1769472 | 4 | 300% | 131072 | 128 | 16 |
| Llama 3.2 1B | 16 | 2048 | 8192 | 32 | 8 | 64 | 1235814400 | 212992 | 491520 | 4 | 300% | 32768 | 32 | 4 |
| Qwen2.5 0.5B | 24 | 896 | 4864 | 14 | 2 | 64 derived | 494032768 | 135168 | 414720 | 7 | 600% | 12288 | 12 | 1.5 |
| Qwen2.5 1.5B | 28 | 1536 | 8960 | 12 | 2 | 128 derived | 1543714304 | 272384 | 881664 | 6 | 500% | 28672 | 28 | 3.5 |
| Qwen2.5 3B | 36 | 2048 | 11008 | 16 | 2 | 128 derived | 3085938688 | 460800 | 1410048 | 8 | 700% | 36864 | 36 | 4.5 |
| Qwen2.5 7B | 28 | 3584 | 18944 | 28 | 4 | 128 derived | 7615616512 | 630784 | 1892352 | 7 | 600% | 57344 | 56 | 7 |
| Qwen2.5 14B | 48 | 5120 | 13824 | 40 | 8 | 128 derived | 14770033664 | 1572864 | 2727936 | 5 | 400% | 196608 | 192 | 24 |
| Qwen2.5 32B | 64 | 5120 | 27648 | 40 | 8 | 128 derived | 32763876352 | 2097152 | 6291456 | 5 | 400% | 262144 | 256 | 32 |
| Qwen2.5 72B | 80 | 8192 | 29568 | 64 | 8 | 128 derived | 72706203648 | 4096000 | 9062400 | 8 | 700% | 327680 | 320 | 40 |
| Qwen3 8B | 36 | 4096 | 12288 | 32 | 8 | 128 | 8190735360 | 958464 | 1769472 | 4 | 300% | 147456 | 144 | 18 |
| Qwen3 14B | 40 | 5120 | 17408 | 40 | 8 | 128 | 14768307200 | 1310720 | 2703360 | 5 | 400% | 163840 | 160 | 20 |
| Qwen3 32B | 64 | 5120 | 25600 | 64 | 8 | 128 | 32762123264 | 2490368 | 5898240 | 8 | 700% | 262144 | 256 | 32 |
| Mistral 7B v0.1 | 32 | 4096 | 14336 | 32 | 8 | 128 derived | 7241732096 | 851968 | 1769472 | 4 | 300% | 131072 | 128 | 16 |
| Mistral NeMo 12B | 40 | 5120 | 14336 | 32 | 8 | 128 | 12247782400 | 1228800 | 2334720 | 4 | 300% | 163840 | 160 | 20 |
| Mixtral 8x7B v0.1 | 32 | 4096 | 14336 | 32 | 8 | 128 derived | 46702792704 | 851968 | Held, per expert MLP (weight index) | 4 | 300% | 131072 | 128 | 16 |
| Phi 3 mini 4k instruct | 32 | 3072 | 8192 | 32 | 32 | 96 derived | 3821079552 | Held, fused qkv_proj (weight index) | Held, fused gate_up_proj | 1 | 0% | 393216 | 384 | 48 |
| Phi 4 | 40 | 5120 | 17920 | 40 | 10 | 128 derived | 14659507200 | Held, fused qkv_proj (weight index) | Held, fused gate_up_proj | 4 | 300% | 204800 | 200 | 25 |
| OLMo 2 7B | 32 | 4096 | 11008 | 32 | 32 | 128 derived | 7298617344 | 1048576 | 1449984 | 1 | 0% | 524288 | 512 | 64 |
| SmolLM2 1.7B | 24 | 2048 | 8192 | 32 | 32 | 64 derived | 1711376384 | 393216 | 737280 | 1 | 0% | 196608 | 192 | 24 |
Where head_dim is not published as its own key it is shown as derived, meaning hidden_size divided by num_attention_heads. The KV columns are inference figures, carried here because adapters change nothing about them and the same head ratio governs both. For whole model and inference sizing use the companion LLM memory calculator and the KV cache size calculator.
Where the formula refuses to answer
Phi 3 mini fuses the query, key and value projections into a single qkv_proj weight and fuses the feed forward gate and up projections into a single gate_up_proj weight (weight index). Phi 4 does the same on both counts (weight index). An adapter on a fused matrix has a different shape from adapters on the parts, so neither target set is published for those two.
Mixtral 8x7B replicates its feed forward weights per expert, with eight experts indexed from 0 to 7 in every layer under block_sparse_moe.experts (weight index), and its config sets num_local_experts to 8 (config.json). One per layer MLP term would undercount an MLP targeted adapter there by the expert count, so that cell is held. Its attention set is published, because Mixtral keeps the standard separate q_proj, k_proj, v_proj and o_proj (weight index). Llama 3.1 8B names those four separately too, along with gate_proj, up_proj and down_proj (weight index), as does Qwen2.5 7B (weight index). Those are the module names the copy ready config emits.
Select a held combination and the planner refuses the whole panel and prints the reason instead of a total. A withheld term never becomes a zero inside a sum you would act on.
Alpha does not cost memory
Raising lora_alpha costs nothing in VRAM and nothing in adapter size. LoRA scales the update by alpha divided by r, where alpha is a constant in r, and with Adam tuning alpha is roughly the same as tuning the learning rate (LoRA). The authors set alpha to the first rank they try and do not tune it further, which reduces the need to retune when the rank is varied (LoRA). The adapter starts as a no op, since A is initialised from a random Gaussian and B from zero, so the update is zero at the start of training (LoRA). Rank and the target set move memory. Sequence length and microbatch size move the activation term only. Dropout, the learning rate, the schedule and the epoch count move the result, not the bill.
The saving comes from the frozen base. LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, which greatly reduces the number of trainable parameters for a downstream task (LoRA abstract), and the original weight matrix remains frozen and receives no further adjustment (PEFT guide). For a large Transformer trained with Adam, VRAM drops by up to two thirds when r is much smaller than d_model, because the Adam state for the frozen parameters does not have to be stored (LoRA). On GPT-3 175B the reported training VRAM went from 1.2TB to 350GB, and at rank 4 with only the query and value projections adapted the checkpoint went from 350GB to 35MB (LoRA). The abstract reports 10,000 times fewer trainable parameters and a 3 times smaller GPU memory requirement against Adam full fine tuning of that model (LoRA abstract), and the paper reports a 25 percent speedup during training on GPT-3 175B compared with full fine tuning (LoRA). Merging matters after training, since LoRA adds no additional inference latency (LoRA abstract) because the adapter weights can be merged with the base model (PEFT guide).
A worked example on Llama 3.1 8B
Llama 3.1 8B sets hidden_size to 4096, num_attention_heads to 32, num_key_value_heads to 8, intermediate_size to 14336 and num_hidden_layers to 32 (config.json). With 32 query heads sharing 8 key and value heads, k_proj and v_proj are a quarter the width of q_proj, a derived consequence of the shared head arrangement grouped query attention describes (GQA). Per layer, q and o each cost 4096 plus 4096 per unit of rank while k and v each cost 4096 plus 1024. Multiply by 32 layers and you land on the q k v o coefficient of 851968 in the table, and the gate up down coefficient of 1769472 the same way from three times 4096 plus 14336.
The same arithmetic reproduces the LoRA paper's own reported runs on GPT-3 175B, where the q and v set gives the 4.7M trainable parameter row of Table 4 at rank 1 and the 37.7M row at rank 8 (LoRA Table 4). Recovering the paper's published counts from the shape is the check that the coefficient is right.
Two presets deserve a second look before you trust an activation figure. Qwen3 32B sets num_attention_heads to 64 and head_dim to 128 against a hidden_size of 5120 (config.json), and Mistral NeMo sets num_attention_heads to 32 and head_dim to 128 against a hidden_size of 5120 (config.json). On Qwen3 32B heads times head_dim comes to 8192, which is larger than its hidden size, and on Mistral NeMo it comes to 4096, which is smaller. Both are derived observations. The adapter count uses the projection width either way, while the activation equation uses hidden_size as its h, so on these two the two figures are sized from different widths and the activation number is the rougher of the pair.
The formulas and where they come from
Adapter parameters, r times the quantity in plus out summed over every targeted projection in every layer, from the rank and shape dependence stated by the LoRA paper (LoRA) and PEFT (PEFT guide). Trainable memory, 2 bytes for the fp16 parameter copy, 2 for the fp16 gradients and 12 for the Adam state made of an fp32 parameter copy plus the momentum and the variance at 4 bytes each, which ZeRO gives as 16 bytes per parameter in total (ZeRO). The momentum and variance alone are the 8 bytes per parameter the Transformers docs state for Adam, alongside 4 bytes per parameter for fp32 gradients and 6 bytes per parameter for keeping both fp32 and fp16 weights (memory anatomy).
Activation memory is an estimate. Korthikanti et al. give the activations of a single transformer layer as s times b times h times the quantity 34 plus 5 times a times s over h, where s is the sequence length, b the microbatch size, h the hidden size, a the attention head count and L the layer count (Reducing Activation Recomputation, with those definitions). That equation is stated for the case where no form of model parallelism is applied (same paper), which is the assumption the planner carries. Full activation recomputation, which recomputes the required activations with an extra forward pass during back-propagation, reduces the total required memory for activations to 2sbhL (same paper), and that is the second option in the selector. Selective recomputation sits between them, reducing the memory needed to store activations to 34 times sbhL divided by the tensor parallel size (same paper), and the paper reports it saving 70 percent and 65 percent of the required memory for activations for the GPT-3 and MT-NLG models respectively (same paper). Recomputation is not free, since the extra forward pass can introduce as much as 30 to 40 percent computational time overhead (same paper). Forward activations are cached for the backward pass and vary in size with batch size, sequence length, model depth and hidden size (memory anatomy), which is why the planner asks for all four.
The 4 bit column. QLoRA reduces memory enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance (QLoRA abstract), which the paper states as going from more than 780GB of GPU memory to less than 48GB without degrading runtime or predictive performance against a 16-bit fully finetuned baseline (QLoRA). PEFT describes the method as quantizing a model to 4 bits and then training it with LoRA (PEFT quantization). NF4 is a data type the authors describe as information theoretically optimal for normally distributed weights (QLoRA). The quantization constants are not free either. With 32-bit constants and a blocksize of 64 they add 0.5 bits per parameter on average before Double Quantization, and Double Quantization saves an average of about 0.37 bits per parameter, roughly 3 GB for a 65B model (QLoRA). The checkbox subtracts the second from the first, a derived remainder, and adds what is left to the 4 bit column. As a sanity check on the shape of the answer, for a 7B LLaMA model on FLAN v2 at batch size 1 the paper measured the LoRA input gradients at 567 MB while the LoRA parameters take up only 26 MB, with the 4-bit base model consuming 5,048 MB, and with gradient checkpointing the input gradients reduce to an average of 18 MB per sequence (QLoRA). Paging the Adam state through NVIDIA unified memory avoids the gradient checkpointing memory spikes that occur when a mini-batch with a long sequence length is processed (QLoRA), worth knowing when a run dies at a long sequence rather than in steady state.
Card capacities. The A100 table lists 80GB of HBM2e GPU memory and NVIDIA also publishes a 40GB PCIe variant (A100). The H100 table lists 80GB for the SXM configuration and 94GB for the NVL configuration (H100). The GeForce RTX 4090 comes with 24 GB of GDDR6X memory (RTX 4090), and the L40S table lists 48GB of GDDR6 with ECC (L40S). A card whose capacity cannot be cited to one of those pages is left out.
Common questions
- Should alpha be twice the rank?
- The paper does not say so. LoRA scales the update by alpha over r and the authors set alpha to the first rank they try without tuning it further (LoRA), which is alpha equal to r. QLoRA went the other way with r 64 and alpha 16 (QLoRA). Pick one and remember it pulls on the same lever as the learning rate under Adam.
- Does a higher rank make the model better?
- QLoRA reports that LoRA rank is unrelated to final performance when LoRA is used on all layers, and that the most critical LoRA hyperparameter is how many adapters are used in total (QLoRA). Coverage first, then rank.
- Which modules should target_modules name?
- In Transformer models LoRA is typically applied to attention blocks only, for simplicity and further parameter efficiency (PEFT guide), and in most experiments the LoRA paper applies the adapter only to the query and value projection matrices (LoRA). PEFT also accepts a string, in which case a regex match is performed (PEFT reference). Tick the boxes and watch the count move before you commit.
- Why does q and v only cost different amounts on different models?
- Because grouped query attention divides query heads into groups, each of which shares a single key head and value head (GQA), so v_proj shrinks with the head ratio while q_proj does not. With a single group the arrangement is equivalent to multi-query attention, and with groups equal to the head count it is equivalent to multi-head attention (GQA). OLMo 2 7B sets num_key_value_heads to 32 (config.json) while Qwen2.5 0.5B sets it to 2 (config.json). Same recipe, very different bill.
- My model is not in the list.
- Pick Custom model and type num_hidden_layers, hidden_size, intermediate_size, num_attention_heads, num_key_value_heads, head_dim and the parameter count. All seven are in the model's own config.json and model card, and a custom entry is treated exactly like a preset.
Where every number on this page comes from
This is how we know, and how you can check it. Every per model number here is fetched, not typed. Parameter counts are read from each model's Hugging Face model-info record, the safetensors.total field, and the adapter coefficients and head ratios are recomputed in code from the shapes published in each model's own config.json, linked on every row. Each value is cross checked against a second, independently operated host, and a value the two hosts do not agree on is held rather than published, which is why some cells read Held. Everything the planner shows you is then computed in your browser from your inputs, using the formulas above and the constants named beside each result. Nothing is sent to a server.
- Estimates use standard published formulas. Real results vary with your data, your settings, and your runtime.
- This tool is for planning and teaching. Check a result against your own measurement before you rely on it.
- All computation runs client side. No data leaves your browser.
ml0x publishes free machine learning calculators and explainers. Every number on this page is computed in your browser from the inputs you enter. Nothing is sent to a server. Written by Michael Lip, part of the Zovo Tools network.