Repository navigation
refactor(losses): compute the regularization loss over regularized layers only - #12
Merged
Merged
Conversation
…yers only Loss.regularization_loss() looped over every trainable layer each step and cast its weights and biases to fp32. Params are always fp32 (PARAM_DTYPE), so the casts did nothing, and on the CIFAR-10 benchmark only one of 11 layers is regularized. remember_trainable_layers(), called once from finalize(), now stores regularized_layers: the trainable layers with a non-zero regularizer. regularization_loss() loops over those and drops the casts. Model.train() reads has_reg from that list instead of repeating the four checks. Adds a hand-computed regularization_loss() test, and makes test_train_with_regularization check has_reg instead of only running.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Loss.regularization_loss()walked every trainable layer each step and cast each one'sweights and biases to fp32, though params are always fp32 and most models regularize only a
layer or two.
Model.train()also had its own copy of the "is this layer regularized" check.The loss now picks out the regularized layers once, at
finalize(), andtrain()reads thatlist instead of repeating the check. One commit.
Regularization loss
Before: on the CIFAR-10 benchmark only conv 08 sets a regularizer (
l2=1e-5), but each stepregularization_loss()looped over all 11 trainable layers. That's 22astype(float32, copy=False)calls, plus about 44 comparisons. The casts never did anything:PARAM_DTYPEis fixed to'float32', and a cast trace showed every call got fp32 input.Now:
Loss.remember_trainable_layers()storesregularized_layers: the trainable layers with atleast one non-zero regularizer. It's called once, from
finalize().regularization_loss()loops over only those layers and useslayer.weights/layer.biasesdirectly. The four> 0checks inside the loop stay, so the value is unchanged.Model.train()getshas_regfrombool(getattr(loss, "regularized_layers", ())). Before,it repeated the four checks over every layer.
This saves host time only.
benchmarks/cifar10_breakdown.py --repeats 3on an RX 9070 XT(gfx1201), against
main:loss.regularization_losshost time: 34.5 → 26.0 µs per step. That's −25%, beyond therun-to-run spread, but below
compare.py's 0.01 ms floor.model.train(): 8.31 → 8.30 ms/step (−0.1%, noise). Training is GPU-bound (about 8.8 ms ofGPU time against 2.0 ms of host time), so the saving doesn't show here. It's a cleanup that
would matter more for small or host-bound models.
Regularizers are only set in layer constructors, so the list can't go stale today. If
something later changes regularizers after
finalize()(a regularization schedule, say), ithas to refresh the list.
Behavior changes
Loss.trainable_layersis gone; onlyregularization_loss()used it. It's replaced byLoss.regularized_layers.remember_trainable_layers()without callingsuper()nowreports no regularization loss in
train(). Before,train()checked the layers itself.The optimizer still applies the regularization gradients either way.
Testing
python3 -m unittest discover tests: 1115 tests pass on NumPy and CuPy (54 skipped), in theROCm devcontainer.
test_regularization_loss_known_valueschecks a hand-computed total of 4.05:Removing any one of the four conditions from the filter makes it fail.
test_train_with_regularizationonly checked that training runs. It now checks thattrain()turnshas_regon for a regularized model and off for an unregularized one.Hardcoding
has_regtoTrueor toFalsemakes it fail. Before, no test caught either.