Follow the low-rank path
Frozen base plus a trainable branch with explicit shapes.
LoRA Rank and Adapter Anatomy
Train the update. Keep the base.
Change a layer’s dimensions and adapter rank. Follow the low-rank path and count exactly which parameters you train.
Live result
One frozen path. One learned path.
Scroll the diagram sideways. Exact values appear below.
Original LoRA with scaling alpha/r, unmerged adapters, and no bias training. Counts are tensor payloads, not training VRAM or checkpoint file size. They exclude gradients, optimizer states, activations, other layers, and metadata. Rank is not a quality score.
The explanation
Two small matrices describe a large update.
A frozen base matrix W0 has d rows and k columns. A maps the k-dimensional input into r coordinates; B maps those coordinates back into d outputs. The update BA is added to W0, rather than replacing it. [L1]
That gives rk + dr trainable values per targeted matrix, compared with dk base values. Multiply by the number of identical targets to obtain the scope shown here. The fraction is adapter parameters divided by selected base parameters, not all model parameters.
Increasing rank grows storage linearly. When r reaches dk/(d+k), parameter savings disappear. The product rank cannot exceed min(r,d,k), and the actual learned rank can be lower.
Original LoRA scales the update by alpha/r. Alpha changes the multiplier without adding parameters. At standard initialization B is zero, so the initial update is zero even though A has values. The worked numerical example uses a hypothetical learned B. [L2]
y = W0*x + (alpha/r)*B*A*x
Open Asset Factory
Take the explanation with you.
Three independent diagrams. Editable source and high-resolution PNGs, with assumptions printed on the image.
Parameters are not VRAM
What the parameter counter includes and leaves out.
Sources checked 2026-09-07
Claims you can inspect.
Formula derivations and teaching assumptions are documented separately.
Download claim ledgerThe original LoRA paper freezes W0 and learns A and B with an additive BA update, scaled by alpha/r.
Primary source ↗No universal quality or memory-saving factor is transferred from the paper’s experiments.
PEFT documents rank, target modules, alpha, bias options, initialization, and variants; default A is initialized and B starts at zero.
Primary source ↗This page selects the original formulation rather than modeling every PEFT option.