forked from Karylab-cklius/vllm
Skip compile SP pass under enforce_eager (V2 runner)
sp_threshold force-enabled compilation_config.pass_config.enable_sp, which the
V2 model runner rejects ("does not yet support: sequence parallelism"). The
eager in-forward SP path reads sp_threshold directly and needs no compile pass;
under enforce_eager there is no compilation to attach it to. Guard the mapping.
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
This commit is contained in:
+5
-1
@@ -1206,7 +1206,11 @@ class VllmConfig:
|
||||
# None -> defer to the optimization-level default
|
||||
# N > 0 -> force the pass on with N as its min-token threshold
|
||||
sp_threshold = self.parallel_config.sp_threshold
|
||||
if sp_threshold is not None:
|
||||
# Only enable the compile-based SP pass when compilation actually runs.
|
||||
# The eager in-forward SP path reads sp_threshold directly and runs the
|
||||
# V2 model runner, which rejects the compile SP pass; under enforce_eager
|
||||
# there is no compilation to attach it to anyway.
|
||||
if sp_threshold is not None and not self.model_config.enforce_eager:
|
||||
pass_config.enable_sp = True
|
||||
if pass_config.sp_min_token_num is None:
|
||||
pass_config.sp_min_token_num = sp_threshold
|
||||
|
||||
Reference in New Issue
Block a user