Summary
local_window_attention defaults block = min(max(64, window), seq). For wide windows the query-block height tracks the window, so the score tensor (batch, heads, block, key_span) becomes near/full sequence-square and can OOM — defeating the blockwise design meant to avoid a full n×n score matrix.
Reproduce (shape math)
window=65536, seq=131072, default block_size=None → block=65536. First scores ≈ (1, 8, 65536, 131072) fp16 ≈ 128 GiB.
Suggested fix
Cap the automatic query block independently of window (e.g. min(64, seq)). Preserve explicit block_size.
Open zone: attention/ only.
Summary
local_window_attentiondefaultsblock = min(max(64, window), seq). For wide windows the query-block height tracks the window, so the score tensor(batch, heads, block, key_span)becomes near/full sequence-square and can OOM — defeating the blockwise design meant to avoid a fulln×nscore matrix.Reproduce (shape math)
window=65536,seq=131072, defaultblock_size=None→block=65536. First scores ≈(1, 8, 65536, 131072)fp16 ≈ 128 GiB.Suggested fix
Cap the automatic query block independently of
window(e.g.min(64, seq)). Preserve explicitblock_size.Open zone:
attention/only.