Skip to content

Record: GDN-Hybrid + Sliding Window Attention (cold-cache, 1.01710 BPB) #1622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
joshkmartinez wants to merge 1 commit into from
Closed

Record: GDN-Hybrid + Sliding Window Attention (cold-cache, 1.01710 BPB) #1622

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions ...rack_10min_16mb/2026-04-11_GDN_Hybrid_DeltaRule_ColdCache3Seed_1.0171/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# GDN-Hybrid + Sliding Window Attention (cold-cache 3-seed mean 1.01710033 BPB)

Joshua-owned SAFE_SUBMISSION confirmation run for the GDN-Hybrid family, reconciled from pulled TensorPool artifacts for `run039-safe019` / `j-xpv7d0b38j`.

## Headline result

- **SAFE_SUBMISSION authority:** `quantized_bpb`
- **3-seed mean:** **1.01710033 BPB**
- **3-seed std:** **0.00133490 BPB**
- **Best seed:** **1.016192 BPB**
- **Worst seed:** `1.018633 BPB`
- **Artifact size range:** `15,522,111` to `15,981,262` bytes
- **Legality lane:** **SAFE_SUBMISSION** — fixed-predictor / no-TTT Track-A; all pulled artifacts stayed below the `16,000,000` byte cap

## Per-seed authoritative results

| Seed | Steps | EMA BPB | Quantized BPB | XSA BPB | Artifact bytes |
|------|------:|--------:|--------------:|--------:|---------------:|
| 314 | 2223 | 1.007670 | 1.016476 | 1.020950 | 15,522,111 |
| 777 | 2239 | 1.007192 | 1.016192 | 1.020919 | 15,814,260 |
| 2718 | 2240 | 1.009535 | 1.018633 | 1.023874 | 15,981,262 |
| **Mean** | — | **1.008132** | **1.01710033** | **1.021914** | **15,772,544.33** |
| **Std (sample)** | — | — | **0.00133490** | — | — |

## Why this matters

- Improves Joshua's prior staged 3-seed SAFE_SUBMISSION artifact `run037-safe017` (**1.02045733 BPB**) by **0.00335700 BPB**.
- Confirms the GDN-Hybrid family remains strong under a fresh cold-cache 3-seed confirmation bundle.
- Reproduces the strongest visible clean-lane architecture family on Joshua-owned infrastructure using pulled artifacts as authority.

## Technique stack

1. **SP1024 tokenizer** with a GDN-hybrid backbone (`[GDN×5] → SWA → [GDN×5] → SWA_shared`).
2. **Fixed-predictor / no-TTT Track-A lane** — no eval-time or pre-quant adaptation in the scored artifact.
3. **MuonEq-R + AdamW** training mix, EMA `0.997`, late QAT threshold `0.15`.
4. **GPTQ int6 + zstd-22** packaging.
5. **Sliding-window attention side path** present in-model, but submission authority remains the pulled `quantized_bpb` values above.

## Legality notes

This record is **SAFE_SUBMISSION** because the scored artifact is a fixed int6 model with **no TTT, no SLOT, no RLS, and no eval-time adaptation**. The pulled logs show all three serialized artifacts below the 16 MB cap. XSA telemetry is reported for completeness, but the submission authority for this lane remains `quantized_bpb` from the pulled artifacts.

## Provenance

- **TensorPool job:** `j-xpv7d0b38j`
- **Canonical bundle:** `~/parameter-golf-project/jobs/run039-safe019/`
- **Pulled artifacts:** `~/parameter-golf-project/state/tp-pulls/run039-safe019/artifacts/`
- **Upstream source PR:** `openai/parameter-golf#1545`
25 changes: 25 additions & 0 deletions ...ack_10min_16mb/2026-04-11_GDN_Hybrid_DeltaRule_ColdCache3Seed_1.0171/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Results provenance for `run039-safe019`

These numbers were reconciled from pulled TensorPool artifacts, not from live bootstrap telemetry.

- **Job ID:** `j-xpv7d0b38j`
- **Canonical bundle:** `~/parameter-golf-project/jobs/run039-safe019/`
- **Pulled artifacts:** `~/parameter-golf-project/state/tp-pulls/run039-safe019/artifacts/`
- **SAFE_SUBMISSION authority:** `quantized_bpb`
- **Lane:** `SAFE_SUBMISSION` (Track-A / fixed predictor / no-TTT)

## Authoritative pulled metrics

| Seed | EMA BPB | Quantized BPB | XSA BPB | Artifact bytes |
|------|--------:|--------------:|--------:|---------------:|
| 314 | 1.007670 | 1.016476 | 1.020950 | 15,522,111 |
| 777 | 1.007192 | 1.016192 | 1.020919 | 15,814,260 |
| 2718 | 1.009535 | 1.018633 | 1.023874 | 15,981,262 |
| **Mean** | **1.008132** | **1.01710033** | **1.021914** | **15,772,544.33** |
| **Std (sample)** | — | **0.00133490** | — | — |

## Notes

1. `submission.json` in the pulled artifact directory carries the authoritative `quantized_bpb` mean (1.01710033) and artifact sizes for this cold-cache confirmation run.
2. Compared with Joshua's prior staged 3-seed GDN-Hybrid SAFE_SUBMISSION artifact `run037-safe017` at `1.02045733 BPB`, `run039-safe019` improves the mean by **0.00335700 BPB**.
3. All three artifacts stayed under the 16,000,000-byte cap, so this candidate is clean-lane submittable.
Loading