Skip to content

perf(llama32_1b): log-depth tree release for the uberkernel barrier - #228

Open
karabambus wants to merge 1 commit into
aifoundry-org:mainfrom
karabambus:llama32-tree-barrier-submission
Open

perf(llama32_1b): log-depth tree release for the uberkernel barrier#228
karabambus wants to merge 1 commit into
aifoundry-org:mainfrom
karabambus:llama32-tree-barrier-submission

Conversation

@karabambus

@karabambus karabambus commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Log-depth release for the uberkernel's chip-wide barrier.

Under batched dispatch the whole decode graph runs as one launch and
uber_barrier_global() executes before every instruction — ~299 times per token.
Its release is serial: one hart issues two fcc_send stores per active shire (64
stores at 32 shires) while ~2048 harts block in fcc_consume.

Change. Each active shire gets a rank = its index among the set bits of the
launch mask. The last arriver credits rank 0 only; each shire's designated hart
(local index 0) forwards to ranks K*r+1 .. K*r+K on waking, before doing any
other work. K is UBER_TREE_FANOUT, default 4.

Critical path at 32 shires: 8 stores best case, 26 worst case, versus 64 flat.

Properties.

  • Credits only shires present in the launch mask, so no stray credit can persist
    in an idle shire's FCC counter and be consumed by the next kernel scheduled
    there. The master shire is excluded structurally by the same rule.
  • Arrivals and releases balance by construction: every active shire is credited
    exactly once, so every hart consumes exactly one credit per barrier.
  • Counter reset precedes fan-out, so a released hart re-entering the barrier
    finds zero.
  • Fan-out arithmetic is K-agnostic; retuning UBER_TREE_FANOUT needs no other
    change.

Verification (local).

  • 51,321 launch masks checked for credit-exactly-once, no credit outside the
    mask, and full reachability, across K ∈ {2,3,4,5,8,16,64} —
    tools/uber_tree_barrier_check.py.
  • Backend gate 13/13, matching the unmodified base.
  • Fan-out is genuinely exercised: sysemu launches 32 shires, and a control build
    with forwarding disabled hangs where this passes.
  • Disassembly confirms the child credit stores follow csrw fcc.
  • No banned instructions, no relocations, .bss empty, barrier counter in
    .data, stack chain 3216 B against a 4160 B budget.
  • mul_mat_Q8_0.c and platform.h byte-identical to the validated revision.

Board result. Decode 17.8109 tok/s at PPL 15.3098 (CPU 15.31, ET-vs-CPU
1.3e-05), samples [17.8673, 17.7664, 17.8109], CV 0.23%. Prefill 79.5405 tok/s.
Against the same build with the flat release — identical in every other respect —
this is +1.59% (17.5318 -> 17.8109).

Builds on the batched-dispatch mechanism from the upstream ET backend
(llama.cpp #24179). Developed with AI assistance.

@github-actions github-actions Bot added track: week-2-challenge Week 2 focused hardware challenge track: llama-3.2-1b-performance Global Llama 3.2 1B performance prize labels Jul 26, 2026
@karabambus
karabambus marked this pull request as ready for review July 26, 2026 11:22
@karabambus
karabambus requested a review from AFOliveira as a code owner July 26, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

track: llama-3.2-1b-performance Global Llama 3.2 1B performance prize track: week-2-challenge Week 2 focused hardware challenge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant