forked from Karylab-cklius/vllm
[Model] M3 MSA indexer: unify top-k buffer, drop numpy, GPU-only decode plan
Follow-ups on the cudagraph-capturable MSA indexer: - Top-k: both decode and prefill now write into the single shared, persistent topk_indices_buffer (decode at [:, :nd], prefill at [:, nd:]) and return views into it -- no fresh per-step top-k allocations. - Build the decode plan + flat page table entirely with torch on-GPU: drop numpy and CpuGpuBuffer; segment offsets/lengths are computed via torch.cumsum into the persistent int32 buffers, and the request-major page table is scattered into the buffer via the on-GPU page indptr (the run bounds reads by indptr, so the full buffer is passed and no host page count is needed). - No GPU->CPU sync on the decode path: scalars come from host ints (num_decode_tokens // num_decodes), and seq_lens.cpu() is confined to the eager prefill branch. The impl forward (fmha OnlyScore + Triton top-k) was already sync-free. test_msa_indexer_impl_matches_triton now also asserts both outputs are views into the persistent buffer. 42/42 in test_minimax_m3.py pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Yongye Zhu <yongye@inferact.ai> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
54810663ff
commit
d2fbaf73c1
@@ -475,6 +475,13 @@ def test_msa_indexer_impl_matches_triton(topk, monkeypatch):
|
||||
msa_impl.index_cache.kv_cache = index_cache
|
||||
triton_impl.index_cache.kv_cache = index_cache
|
||||
|
||||
# Exercise the shared persistent top-k buffer: the MSA impl must write both
|
||||
# decode ([:, :nd]) and prefill ([:, nd:]) into it and return views of it.
|
||||
nd = sum(q for q in batch.query_lens if q <= 1)
|
||||
msa_impl.topk_indices_buffer = torch.full(
|
||||
(num_idx_heads, num_tokens, topk), -2, dtype=torch.int32, device=device
|
||||
)
|
||||
|
||||
attn_metadata = {
|
||||
msa_impl.index_cache.prefix: msa_builder.build(0, common),
|
||||
triton_impl.index_cache.prefix: triton_builder.build(0, common),
|
||||
@@ -487,6 +494,10 @@ def test_msa_indexer_impl_matches_triton(topk, monkeypatch):
|
||||
assert msa_prefill is not None and tri_prefill is not None
|
||||
_assert_topk_indices_equal_unordered(msa_decode, tri_decode)
|
||||
_assert_topk_indices_equal_unordered(msa_prefill, tri_prefill)
|
||||
# decode/prefill outputs are views into the one persistent buffer.
|
||||
buf = msa_impl.topk_indices_buffer
|
||||
assert msa_decode.data_ptr() == buf[:, :nd, :].data_ptr()
|
||||
assert msa_prefill.data_ptr() == buf[:, nd:, :].data_ptr()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
|
||||
Reference in New Issue
Block a user