[AlexanderSoftCode]

Context

Conv2d's WMMA backward casts dY to fp16, which rounds anything below about 3e-8 to exactly 0, with no warning. Per-element gradients are often smaller than that, and they shrink as image size grows, so accuracy dropped at higher resolution. On the v1 CIFAR-10 CNN (float16, 25 epochs, seeds 42/43/44, RX 9070 XT, nearest-neighbour upscaled CIFAR), a Python prototype of this fix gave:

Setting	Before	Scale 2^10
32px, batch 256 (mean test accuracy)	80.44%	80.37%
64px, batch 256 (mean test accuracy)	53.17%	72.46%

At 64px, up to 95% of nonzero dY in a layer rounded to 0, and cosine against an fp32 backward pass fell as low as 0.31. With 2^10, at most 0.5% rounded to 0, and cosine was 1.0000 on every layer. The unset precision policy binds the same path, so default GPU training is exposed too, not just set_precision('float16').

dY is now multiplied by a fixed 2^10 in the cast (kernel and astype fallback), and dweights, dbiases and dinputs are divided back down in the existing kernel epilogues. There are no new launches, no new Python work on the hot path, and no new option. Gradients stay in true units outside the layer, so the loss, Dense, optimizers and manual loops are unaffected.

One commit.

Behavior changes

• Same-seed training results change on the WMMA path.
• Each dY element is now silently clipped at about +-64 instead of +-65504. fp16's range is fixed, so the clip is always 65504 / scale. The largest |dY| seen was about 1.7e-3, but an unnormalized loss or an exploding gradient could reach it.
• plan_dweight_launch and plan_dinput_launch now expect dY already scaled by GRAD_SCALE. Launching them directly with unscaled dY gives results 1024x too small.

The fp32 fallback path is unchanged.

Testing

• python3 -m unittest discover tests: 1190 tests pass on NumPy and CuPy (78 skipped).
• New test_conv_gpu_backward_tiny_gradients_survive_fp16_cast: dY around 1e-8 against the fp32 fallback, on scalar and vector staging, with and without the cast kernel, plus a hand-computed dbiases check. Confirmed to fail on the old code. Existing parity tests missed this because they all use dY of magnitude about 1.
• tests/ops/test_conv_cast_kernel.py rewritten for the scaled clip, with 2**10 hardcoded in the expected values. Confirmed to fail on the old code.
• Relative error at dY around 1e-8, RX 9070 XT:

Output	Old kernels	New kernels
dweights	102-104%	0.19%
dbiases	92-112%	0.05-0.24%
dinputs	91-100%	0.14-0.17%

• benchmarks/compare.py, 32px, RX 9070 XT: model.train() went from 4.851 to 4.873 ms/step (+0.5%), with no phase flagged as changed.
• RTX 3050: the CuPy variant of the new test passes

Not verified:

• Trained accuracy on the kernel version. The accuracy numbers above come from the Python prototype, which does the same arithmetic.
• A training run with the unset precision policy. It is covered at layer level only.
• 64px at batch 1024, 96px with scaling, and timing above 32px.
• Real high-resolution data. Upscaled CIFAR adds no image detail, so it is a stress test.

Follow-ups (out of scope)

• A fixed scale will eventually fall short at larger sizes or batches, and where that happens is unknown. A per-step scale chosen by a GPU-side max pass would avoid that and remove the lower clip, without a host readback, at the cost of one extra pass per Conv per step. Not prototyped.
• About 8 points remain between 64px (72.46%) and 32px (80.44%). The cause is unproven, though it may be the fixed three-block model seeing less of an upscaled image.
• Model.set_precision docstring: note that under float16 on CuPy only Dense outputs float16, while Conv2d, BatchNorm and the pools output float32.