Repository navigation
[QNN] 16a8w: gathered tensor read wrong by a consumer that only feeds a graph output #24125
Description
🐛 Describe the bug
In a 16a8w HTP graph, a tensor gathered in-graph (table[pos]) is read wrong by its last consumer when that consumer's result only feeds a graph output. Below, cos/sin are gathered once and used by two RoPEs: output 0 (which also feeds a bmm) is correct, output 2 is not (rel err 0.77 vs 0.0017 for the QDQ module on host). Position p gets row p % 40 for most rows.
Not affected: the same graph in fp16, passing cos/sin as inputs instead of gathering them, or a single RoPE. We hit it in a TTS model where the last layer's keys are a method output; returning one more tensor computed from them works around it.
import numpy as np
import torch
from executorch.backends.qualcomm.export_utils import make_quantizer
from executorch.backends.qualcomm.quantizer.quantizer import QuantDtype
from executorch.backends.qualcomm.serialization.qc_schema import QcomChipset
from executorch.backends.qualcomm.utils.utils import (
generate_htp_compiler_spec, generate_qnn_executorch_compiler_spec, to_edge_transform_and_lower_to_qnn)
from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e
H, N, D = 6, 320, 128
def rope(x, cos, sin):
return x * cos + torch.cat([-x[..., D // 2:], x[..., :D // 2]], -1) * sin
class M(torch.nn.Module):
def __init__(self):
super().__init__()
ang = torch.arange(2048).float()[:, None] / 1e6 ** (torch.arange(0, D, 2).float() / D)
self.register_buffer("cos_t", torch.cat([ang.cos(), ang.cos()], -1))
self.register_buffer("sin_t", torch.cat([ang.sin(), ang.sin()], -1))
def forward(self, x0, x1, pos):
cos, sin = self.cos_t[pos.long()], self.sin_t[pos.long()]
k0 = rope(x0, cos, sin)
# k0 also feeds a bmm; k1 feeds only the graph output and comes out wrong
return k0, torch.bmm(k0, x0.transpose(1, 2)), rope(x1, cos, sin)
torch.manual_seed(0)
sample = lambda: (torch.randn(H, N, D) * 3, torch.randn(H, N, D) * 3, torch.arange(N, dtype=torch.int32)) # noqa: E731
m, ex = M().eval(), sample()
q = prepare_pt2e(torch.export.export(m, ex).module(), make_quantizer(quant_dtype=QuantDtype.use_16a8w, soc_model="SM8850"))
for _ in range(8):
q(*sample())
q = convert_pt2e(q)
specs = generate_qnn_executorch_compiler_spec(soc_model=QcomChipset.SM8850,
backend_options=generate_htp_compiler_spec(use_fp16=False))
with open("rope_gather.pte", "wb") as f:
f.write(to_edge_transform_and_lower_to_qnn(q, ex, specs).to_executorch().buffer)
for i, t in enumerate(ex):
t.numpy().tofile(f"input_{i}.raw")
np.save("expected_out2.npy", m(*ex)[2].numpy())
print("QDQ rel err of output 2:", float((q(*ex)[2] - m(*ex)[2]).norm() / m(*ex)[2].norm()))Run rope_gather.pte with the three inputs and compare output 2 with expected_out2.npy.
Versions
ExecuTorch 1.4.1 (our fork of v1.4.1, not yet checked on main), QAIRT 2.47.0.260601, SM8850 (Galaxy S26 Ultra, HTP v81).
cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @qti-horodnic @cbilgin @abhinaykukkadapu @psiddh
Activity
yenhao-huang commentedon Nov 6, 2026
It seems the issue also happen beyond RoPE: in the 16a8w quantization setting, a simple
index+ element-wise mul also gives wrong results on HTP, possibly due to an op-fusion issue.