Sparse-First Fine-Tuning: Full-Weight Training at the Cost of the Active Set

1. Introduction

Full fine-tuning is expensive for a structural reason. Adam keeps two moment estimates for every weight, and mixed-precision training adds a full-precision master copy and a gradient buffer. The result is about 16 bytes per parameter, or about 112 GB for a 7B model before activations. Every one of those bytes is touched on every step.

Most of that work changes nothing. On a given batch, only the embedding rows of the tokens present receive gradient, and only the projection rows with non-negligible activation produce a gradient worth applying. In our fine-tuning samples the active share stays near 1% of all parameters, and rarely goes above it.

The usual responses keep the dense step and shrink what it applies to. Adapter methods freeze the model and train a small low-rank side path. Quantization stores the frozen model in fewer bits. We take the other route: keep training the full weights, but make the step pay only for the parameters that are active. Sparsity is the design principle from the first step, not a compression applied afterward.

2. Method

One training step runs five sparse stages and one coupled update. A parameter that is not active skips all of them.

2.1 Observing activity

Each step produces a per-row binary mask for every weight matrix. For the embedding, a component we call the conductor keeps a charge for every row. At the end of each observation window, every row's charge halves and gains half of its share of the window's hits, the row's hit count divided by the busiest row's:

Row r is active while qᵣ ≥ 0.01. A row seen often stays charged across windows, and a silent row fades below the 1% threshold within a few windows. The conductor runs on the CPU and reads only the batch's token IDs, which are already in host memory. It never asks the GPU anything. For each projection (query, key, value, output, gate, up and down), a row tracker does the same with the output magnitudes the forward pass produced. The masks are computed once a step and reused by every later stage.

2.2 Masked backward pass

The weight gradient of a projection is a transposed product, dW = dXᵀ · A. The masked kernel checks the mask per tile of 32 rows: a tile with no active row returns at once, with no loads, no accumulation and no writes. Inside an active tile, inactive rows are skipped before the final write. At a large vocabulary with a typical batch, this removes almost all of the embedding's gradient work.

2.3 Sparse state and int8 weights

Each weight is stored as an int8 value q with a per-row scale s and a full-precision residual δ that carries the precision below one quantization step:

The first and second moments m and v are kept in fp16, and only for active rows. The update kernel reads q and δ, rebuilds the fp32 weight in registers, applies the update of §2.4, and writes back a new q and δ. The full-precision weight exists only in registers. Inactive rows return immediately. The same mask drives a sparse write-back, which rebuilds the live weights for the next forward pass only where they changed.

2.4 The coupled update

Some weight matrices work as pairs: the gate and up projections of a feed-forward block, and the query–output and key–value projections of attention. Helix updates each pair together, so that each matrix's gradient informs its partner's moments. Every layer has three pairs and no unpaired matrix. A pair has a coupling strength b: 0.6 for the strongly coupled gate–up pair and 0.4 for the attention pairs.

The coupling rotates. A phase φ advances by 36° times a stride on every step (§2.5), and sets how much of the update goes through a primary channel and how much through a secondary one:

Here e is the exposure of the primary channel, M and μ are the primary and secondary channel weights, and α is a fixed pitch angle. g₁ and g₂ are the clipped gradients of the two matrices, and g̃₁, g̃₂ their effective gradients. ψ is a slow non-periodic phase taken from a point walking an elliptic curve, and w₊, w₋ are the channel weights of §2.5. Each effective gradient then drives a standard Adam step with bias correction. The weight decay is scaled by cos α, the stable component of the pitch:

A per-parameter optimizer such as AdamW only moves along coordinate axes. The cross terms let a step move along a direction that encodes the functional relationship between the two matrices. Unpaired parameters, such as norms, take a plain Adam step.

2.5 Adapting to the loss trend

Helix keeps a short history of loss changes. Their exponentially weighted mean, with a decay length of ten steps, measures how consistently training is descending:

Here ΔLᵢ is the change in loss i steps ago, so a positive C means the loss is falling.

When the loss falls steadily, the primary channel dominates. On a plateau the secondary channel grows, which shifts the update toward exploration. A smoothed copy of C also sets the stride by which the phase advances. While it stays above a small positive threshold, the stride walks up the Fibonacci sequence (1, 1, 2, 3, 5 …, capped at 21), so the rotation covers more ground per step. When it turns negative, the stride steps back down. Slower Fibonacci counters at the scale of a full turn and of ten turns add a gentle modulation of ±10% and ±5% to the channel weights.

2.6 Checkpoint and rewind

The optimizer keeps a checkpoint of the active rows whenever the loss is within 10% of the last checkpoint. A new loss minimum opens a ten-step watch. If at the end of the watch the loss has risen more than 15% above that minimum, Helix restores the checkpoint, resets the stride to 1 and skips the step; it allows three such recoveries per minimum. Otherwise the minimum is accepted and becomes the new checkpoint. Because only active rows are saved, a checkpoint costs no more than the active set.

The rewind changes how aggressively the optimizer can move. A dense optimizer must keep its steps conservative, because a diverged run is lost. Helix can take large steps, knowing that an overshoot costs a few steps, not the run.

Figure 2 shows how one step is prepared. Everything the CPU decides for step N reads only what step N − 1 has already left in shared memory: the loss, the activations and the hot rows. The conductor reads the batch's token IDs, which are on the CPU anyway. Nothing is copied back from the GPU in the middle of a step, so the readback never shows on the GPU's clock. The price is a learning signal one step old, which the rewind and the decaying windows absorb.

2.7 Several GPUs without gradient synchronization

Data-parallel training replicates the model and averages the full gradient across GPUs on every step. Sparse-first training splits sequence positions instead: with N GPUs, GPU g computes the positions where p mod N = g. Each GPU holds the full weights and runs every matrix product on its share of the rows. The only exchange per layer is the keys and values that causal attention needs, about 3 MB a step at a model width of 4,096. Each GPU runs its own optimizer, and the shared activity masks keep the copies aligned without a gradient all-reduce.

2.8 Design choices

Why a rotating coupling, not a fixed one? A fixed coupling picks one blend of the two matrices' gradients and applies it forever, which biases every step toward the same direction. Rotating the blend sweeps each pair through the whole range, from mostly-own-gradient to strongly-shared, over each turn. No single coupling is privileged. Each step is still an Adam step; the rotation only decides what gradient it sees.

Why this rotation and not another? We tried many. Among them: a sinusoidal swing of the step size, a square-wave alternation between a high and a low step, a "screw" whose swing decays as training goes on, a horizontal screw, and vibration-style perturbations. None beat the original rotating coupling.

Why two channels with unequal weight? One channel follows the descent and one explores. Giving them unequal weight (1.833 to 1) keeps descent dominant while the loss falls, and lets exploration grow only on a plateau, which is when it pays.

Why a Fibonacci stride, not doubling or a fixed schedule? The stride should grow while training goes well and collapse at once when it does not. Fibonacci growth is geometric, so it accelerates, but its ratio tends to 1.618, much gentler than doubling, so the stride never jumps past the region it was just rewarded for. Every step is a whole number of rungs, the sequence restarts cleanly from 1 after a rewind, and the cap of 21 keeps one step to about two turns. A cosine or step schedule follows the clock. This one follows the loss.

Why a non-periodic phase term? A purely periodic modulation can fall into step with periodic structure in the data, such as a fixed batch order. The small phase term from the elliptic-curve walk never repeats, so the rotation cannot lock onto a data period.

Why rewind, not a lower learning rate? A lower learning rate pays for safety on every step. A rewind pays only when a step actually overshoots, and it costs a few steps of the active set. That trade lets the optimizer keep a large step size.

Why the names? The geometry came from B-form DNA: ten rungs a turn (36° each), a 28.4° pitch, a major and a minor groove in a ratio of about 1.833, and pairs bonded more or less tightly. The analogy gave us a coherent set of constants to start from. The constants are what matter, and every one of them is in the code.

3. Memory

The training-state memory of a model with N parameters, of which a share h is active, is:

Here b_base is the bytes per parameter every method pays, and b_active the bytes per active parameter:

• Full fine-tuning (AdamW): b_base = 16: 2 for the weight, 4 for the master copy, 8 for the moments and 2 for the gradient.
• LoRA, rank 16: b_base = 2, plus about 40M adapter parameters at 16 bytes each.
• QLoRA: b_base ≈ 0.52, plus the same adapters.
• Helix: b_base = 1 and b_active = 12: 4 for the residual, 2 + 2 for the moments and 4 for the gradient.

For a 7B model with h = 1%, the Helix total is 7 GB + 0.84 GB.

Table 1. The values behind Figure 3.

QLoRA uses less memory than Helix, because its frozen base is stored in 4 bits and ours in 8. No amount of sparsity closes that gap. What sparse-first training offers is a different trade: training of the full weights at about half of LoRA's memory, and about one-fourteenth of full fine-tuning.

4. Results

4.1 Throughput

We measured training steps per second against plain PyTorch on the same model shapes: a byte-level transformer with four layers, a sequence length of 64 and a vocabulary of 256. On the Apple M4 Max the baseline is PyTorch on MPS. On two H100 SXM GPUs it is PyTorch DDP, against the position parallelism of §2.7.

Table 2. The values behind Figure 4.

The one slower size, 38M, came from our tiled matrix kernels at that shape, which lost to PyTorch's fused operators there; it is not a property of the method. These runs used our earlier GPU backend. Our runtime has become faster since, so the figures are conservative.

4.2 Convergence

Figure 5 follows one run on an RTX 5090: a model width of 512, four layers, 500 steps. The loss falls from 6.17 to 1.95, a 4.8x drop. The best-loss floor shows the rewind at work: at step 356 the loss rebounded past the threshold, Helix restored the step-336 checkpoint, and the floor dropped further after the restart.

Table 3. The values behind Figure 5.

5. Related work

Adapters. LoRA [1] freezes the model and trains a low-rank update, which shrinks the optimizer state to the adapter. QLoRA [2] adds a 4-bit frozen base and is the leanest way to fine-tune today. Unsloth [3] builds on it with careful kernels and memory engineering, and shares our efficiency-first ethos; hats off to them. Adapter methods and sparse-first training answer different questions. Adapters make training a side path as lean as possible. Sparse-first training makes training the full weights affordable. We have not raced Unsloth on speed and make no claim there.

Sparsity. The lottery ticket hypothesis [4] shows that sparse subnetworks exist, but finds them through dense training. N:M sparsity [5] removes weights in fixed patterns. Mixture-of-experts models [6] route inputs through a learned gate. Our sparsity is unstructured and dynamic: it is observed from the data on every step, and nothing is pruned or learned to produce it.

Quantization. Quantization-aware training [7] runs a quantized forward pass over a dense full-precision backward pass and optimizer. Here the weights and the optimizer state are both compact, and a full-precision residual keeps the update precision that int8 alone would lose.

6. Limitations

The memory figures of §3 are computed from bytes per parameter, not measured, and they leave out activations. The 1% active share describes our fine-tuning samples; a task that touches more of the model costs more, about 0.84 GB per further percent at 7B. The throughput results use small byte-level models and are compared with plain PyTorch only; we have not compared speed or final quality against LoRA, QLoRA or Unsloth.

7. Conclusion

Dense training is a choice, not a necessity. Most gradients are near zero, most parameters do not move on a given step, and most embedding rows are silent. Sparse-first training builds every stage around that: activity masks, a masked backward pass, sparse optimizer state, int8 weights with a full-precision residual, sparse write-back, and a coupled optimizer that can afford to explore. Work scales with the active set, not the size of the model. Helix is our fine-tuning engine today, and its rule runs through everything the lab has built since: spend memory and compute only where they buy something.

Never multiply by zero.

References

1. Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
2. Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs.
3. Unsloth AI. Unsloth (software).
4. Frankle, J., & Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.
5. Zhou, A., et al. (2021). Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch.
6. Fedus, W., et al. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.
7. Jacob, B., et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.