[Model] M3 MSA indexer: unify top-k buffer, drop numpy, GPU-only decode plan

Follow-ups on the cudagraph-capturable MSA indexer:

- Top-k: both decode and prefill now write into the single shared, persistent
  topk_indices_buffer (decode at [:, :nd], prefill at [:, nd:]) and return views
  into it -- no fresh per-step top-k allocations.
- Build the decode plan + flat page table entirely with torch on-GPU: drop numpy
  and CpuGpuBuffer; segment offsets/lengths are computed via torch.cumsum into the
  persistent int32 buffers, and the request-major page table is scattered into the
  buffer via the on-GPU page indptr (the run bounds reads by indptr, so the full
  buffer is passed and no host page count is needed).
- No GPU->CPU sync on the decode path: scalars come from host ints
  (num_decode_tokens // num_decodes), and seq_lens.cpu() is confined to the eager
  prefill branch. The impl forward (fmha OnlyScore + Triton top-k) was already
  sync-free.

test_msa_indexer_impl_matches_triton now also asserts both outputs are views into
the persistent buffer. 42/42 in test_minimax_m3.py pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Yongye Zhu <yongye@inferact.ai>

Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
This commit is contained in:
Yongye Zhu
2026-06-17 05:32:36 +00:00
co-authored by Claude Opus 4.8
parent 54810663ff
commit d2fbaf73c1
2 changed files with 83 additions and 53 deletions
@@ -475,6 +475,13 @@ def test_msa_indexer_impl_matches_triton(topk, monkeypatch):
msa_impl.index_cache.kv_cache = index_cache
triton_impl.index_cache.kv_cache = index_cache
# Exercise the shared persistent top-k buffer: the MSA impl must write both
# decode ([:, :nd]) and prefill ([:, nd:]) into it and return views of it.
nd = sum(q for q in batch.query_lens if q <= 1)
msa_impl.topk_indices_buffer = torch.full(
(num_idx_heads, num_tokens, topk), -2, dtype=torch.int32, device=device
)
attn_metadata = {
msa_impl.index_cache.prefix: msa_builder.build(0, common),
triton_impl.index_cache.prefix: triton_builder.build(0, common),
@@ -487,6 +494,10 @@ def test_msa_indexer_impl_matches_triton(topk, monkeypatch):
assert msa_prefill is not None and tri_prefill is not None
_assert_topk_indices_equal_unordered(msa_decode, tri_decode)
_assert_topk_indices_equal_unordered(msa_prefill, tri_prefill)
# decode/prefill outputs are views into the one persistent buffer.
buf = msa_impl.topk_indices_buffer
assert msa_decode.data_ptr() == buf[:, :nd, :].data_ptr()
assert msa_prefill.data_ptr() == buf[:, nd:, :].data_ptr()
@pytest.mark.parametrize(