Skip to content

refactor(losses): compute the regularization loss over regularized layers only - #12

Merged
AlexanderSoftCode merged 1 commit into
mainfrom
refactor/loss-regularized-layers
Oct 26, 2026
Merged

AlexanderSoftCode merged 1 commit into
mainfrom
refactor/loss-regularized-layers

Conversation

@AlexanderSoftCode

Copy link
Copy Markdown
Owner

Context

Loss.regularization_loss() walked every trainable layer each step and cast each one's
weights and biases to fp32, though params are always fp32 and most models regularize only a
layer or two. Model.train() also had its own copy of the "is this layer regularized" check.
The loss now picks out the regularized layers once, at finalize(), and train() reads that
list instead of repeating the check. One commit.

Regularization loss

Before: on the CIFAR-10 benchmark only conv 08 sets a regularizer (l2=1e-5), but each step
regularization_loss() looped over all 11 trainable layers. That's 22
astype(float32, copy=False) calls, plus about 44 comparisons. The casts never did anything:
PARAM_DTYPE is fixed to 'float32', and a cast trace showed every call got fp32 input.

Now:

  • Loss.remember_trainable_layers() stores regularized_layers: the trainable layers with at
    least one non-zero regularizer. It's called once, from finalize().
  • regularization_loss() loops over only those layers and uses layer.weights /
    layer.biases directly. The four > 0 checks inside the loop stay, so the value is unchanged.
  • Model.train() gets has_reg from bool(getattr(loss, "regularized_layers", ())). Before,
    it repeated the four checks over every layer.

This saves host time only. benchmarks/cifar10_breakdown.py --repeats 3 on an RX 9070 XT
(gfx1201), against main:

  • loss.regularization_loss host time: 34.5 → 26.0 µs per step. That's −25%, beyond the
    run-to-run spread, but below compare.py's 0.01 ms floor.
  • model.train(): 8.31 → 8.30 ms/step (−0.1%, noise). Training is GPU-bound (about 8.8 ms of
    GPU time against 2.0 ms of host time), so the saving doesn't show here. It's a cleanup that
    would matter more for small or host-bound models.

Regularizers are only set in layer constructors, so the list can't go stale today. If
something later changes regularizers after finalize() (a regularization schedule, say), it
has to refresh the list.

Behavior changes

  • No change to loss values or gradients: the same terms are summed over the same layers.
  • Loss.trainable_layers is gone; only regularization_loss() used it. It's replaced by
    Loss.regularized_layers.
  • A custom loss that overrides remember_trainable_layers() without calling super() now
    reports no regularization loss in train(). Before, train() checked the layers itself.
    The optimizer still applies the regularization gradients either way.

Testing

  • python3 -m unittest discover tests: 1115 tests pass on NumPy and CuPy (54 skipped), in the
    ROCm devcontainer.
  • New test_regularization_loss_known_values checks a hand-computed total of 4.05:
    • four layers, each with exactly one of the four regularizers;
    • one unregularized layer with large weights, which must add nothing.
      Removing any one of the four conditions from the filter makes it fail.
  • test_train_with_regularization only checked that training runs. It now checks that
    train() turns has_reg on for a regularized model and off for an unregularized one.
    Hardcoding has_reg to True or to False makes it fail. Before, no test caught either.

…yers only

Loss.regularization_loss() looped over every trainable layer each step and
cast its weights and biases to fp32. Params are always fp32 (PARAM_DTYPE),
so the casts did nothing, and on the CIFAR-10 benchmark only one of 11
layers is regularized.

remember_trainable_layers(), called once from finalize(), now stores
regularized_layers: the trainable layers with a non-zero regularizer.
regularization_loss() loops over those and drops the casts. Model.train()
reads has_reg from that list instead of repeating the four checks.

Adds a hand-computed regularization_loss() test, and makes
test_train_with_regularization check has_reg instead of only running.
@AlexanderSoftCode
AlexanderSoftCode merged commit 85f016c into main Oct 26, 2026
5 checks passed
@AlexanderSoftCode
AlexanderSoftCode deleted the refactor/loss-regularized-layers branch October 26, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant