+
+
+
+
+ + ${overall.completed} of ${overall.total} items completed +
+${dueCards.length} cards due today
+${allCards.length} total cards in your deck
+Quiz not available yet.
'; + return; + } + + // Check for previous attempts + const history = await DB.getQuizHistory(quiz.weekId, quiz.quizId); + const bestScore = history.length > 0 ? Math.max(...history.map(h => h.score)) : null; + + container.innerHTML = ` ++ ${i + 1}. ${q.question} +
+ + +This unit is coming soon.
+Covers:
+No flashcards yet. Complete Unit 1 to unlock your first deck.
+No cards due for review today. Check back tomorrow.
'} +Card ${idx + 1} / ${cards.length}
+Click to reveal
+ +All cards reviewed for today.
${overall.completed} items completed, ${overall.total} total tracked
++ The mathematical and architectural foundations that every technique in this competition builds on. + Information theory tells us why compression equals prediction. The transformer is the machine that does it. +
++ A language model assigns probabilities to sequences of tokens. Given a context + $x_1, x_2, \ldots, x_{t-1}$, the model predicts a distribution over the next token $x_t$. + The chain rule of probability decomposes any sequence probability as: +
++ $$P(x_1, \ldots, x_T) = \prod_{t=1}^{T} P(x_t \mid x_1, \ldots, x_{t-1})$$ +
++ This is the autoregressive factorization. The entire Parameter Golf challenge reduces to: + build the best next-token predictor that fits in 16MB. +
++ Cross-entropy measures how well our model $q$ approximates the true distribution $p$: +
++ $$H(p, q) = -\sum_{x} p(x) \log q(x)$$ +
++ In practice, we compute the average negative log-likelihood over tokens (in nats when using $\ln$). +
++ Perplexity = $e^{H(p,q)}$ (or $2^{H}$ in bits). Lower is better. +
++ Bits-per-byte (BPB) is the competition metric. It converts token-level loss to byte-level compression: +
++ $$\text{BPB} = \frac{\text{val\_loss}}{\ln 2} \times \frac{\text{tokens}}{\text{bytes}}$$ +
++ This makes the metric tokenizer-agnostic: a model with a 1024-token vocabulary and one with 32K tokens + are compared on the same byte-compression scale. +
++ Shannon entropy $H(X) = -\sum p(x) \log_2 p(x)$ is the theoretical minimum bits needed to encode a source. + The source coding theorem says you cannot compress below $H(X)$ bits per symbol on average. +
++ The deep insight: optimal compression and optimal prediction are the same problem. + A model that perfectly predicts the next token achieves the best possible compression rate. + This is why the Parameter Golf metric (BPB) directly measures compression quality. +
++ KL divergence measures the extra bits needed when using $q$ instead of $p$: +
++ $$D_{KL}(p \| q) = \sum_x p(x) \log \frac{p(x)}{q(x)} = H(p, q) - H(p) \geq 0$$ +
++ Since $H(p)$ is fixed, minimizing cross-entropy $H(p,q)$ is equivalent to minimizing $D_{KL}(p \| q)$. + Our training loss is literally measuring how far our model is from the true distribution. +
++ Start with bigrams (n=2). Count all character pairs in a training text, normalize to get probabilities, + then compute $\text{BPB} = -\frac{1}{N} \sum \log_2 P(c_t | c_{t-1})$ on held-out text. + Use add-1 (Laplace) smoothing for unseen pairs. +
+ ++ Start from the definition of cross-entropy in nats. Convert to bits (divide by $\ln 2$). + Perplexity is $e^{H}$ in nats or $2^{H}$ in bits. For BPB, you need to account for the + tokens-to-bytes ratio of the tokenizer. +
+ ++ Use the Gibbs' inequality: $H(p, q) = H(p) + D_{KL}(p \| q) \geq H(p)$, with equality + iff $q = p$ everywhere. Since $D_{KL} \geq 0$ (prove via Jensen's inequality on $\log$). +
+ ++ The core operation of the transformer. Input $X \in \mathbb{R}^{T \times d}$ is projected into + queries, keys, and values: +
++ $$Q = XW_Q, \quad K = XW_K, \quad V = XW_V$$ +
++ $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$ +
++ The $\sqrt{d_k}$ scaling prevents the dot products from growing too large, keeping softmax + in a well-behaved regime. In the Parameter Golf baseline, $d_k = d_\text{model} / n_\text{heads} = 64$. +
++ RoPE (Rotary Position Embedding) is used in the Parameter Golf baseline. + It encodes position by rotating query and key vectors in 2D subspaces: +
++ $$q_m = R_\Theta(m) \cdot W_q x_m, \quad k_n = R_\Theta(n) \cdot W_k x_n$$ +
++ where $R_\Theta(m)$ applies rotation by angle $m \cdot \theta_i$ to the $i$-th dimension pair. + The attention score $q_m^\top k_n$ depends only on the relative position $m - n$. +
++ The SOTA uses Partial RoPE: only 16 of 64 head dimensions get position encoding. + The other 48 dimensions attend purely by content similarity. +
+
+ The baseline uses ReLU^2: $\text{FFN}(x) = W_2 \cdot (\text{ReLU}(W_1 x))^2$.
+ Squaring after ReLU creates a smoother, sparser activation pattern.
+ The SOTA uses LeakyReLU(0.5)^2 which allows small negative gradients to flow.
+
+ The MLP expansion factor in the baseline is 2x (hidden = 2 * model_dim). + The SOTA uses 3x (hidden = 3 * 512 = 1536) for more capacity per layer. +
+
+ Build it layer by layer: Embedding -> PositionalEncoding ->
+ N x (MultiHeadAttention + FeedForward + LayerNorm + residuals) ->
+ Linear head. Use the causal mask in scaled_dot_product_attention.
+ Reference the baseline train_gpt.py for a clean implementation to compare against.
+
+ Compare your implementation against the baseline's Rotary class and
+ apply_rotary_emb function. Verify that $q_m^\top k_n$ only depends on $m-n$
+ by computing attention scores and checking the Toeplitz structure.
+
+ Use a small text file (Shakespeare, Wikipedia excerpt). Tokenize with a character-level or + tiny BPE vocabulary. Train for ~1000 steps with Adam. Plot the loss curve. + The model should overfit the training set if it's small enough. That's fine, it proves your + architecture works. +
+ ++ Scaling laws quantify how model performance improves with more parameters, data, and compute. + Parameter Golf is an L(N) optimization problem: minimize loss at a fixed parameter count. + Understanding these laws tells you where to spend your 16MB budget. +
++ Kaplan et al. (2020) found that language model loss follows power laws: +
++ $$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \quad L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}$$ +
++ where $N$ = parameters, $D$ = dataset tokens, $C$ = compute (FLOPs). + The exponents are approximately $\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$, $\alpha_C \approx 0.050$. + These are remarkably smooth and predictable across many orders of magnitude. +
++ The key insight: loss is predictable from scale. You can extrapolate from small runs to predict large run performance. +
++ Hoffmann et al. (2022) showed that for a fixed compute budget $C$, you should scale + parameters $N$ and data $D$ roughly equally: $N \propto C^{0.5}$ and $D \propto C^{0.5}$. + Previous practice (GPT-3) over-allocated to parameters and under-allocated to data. +
++ For Parameter Golf, Chinchilla is relevant but not directly applicable: we're constrained on $N$ (16MB artifact), + not on $C$ or $D$. We want to overfit on compute to maximize what we extract from a fixed $N$. +
++ Standard scaling laws assume you're training a standard transformer with optimal hyperparameters. + Parameter Golf breaks this assumption: the architecture is free, quantization is free, evaluation tricks are free. + You're optimizing a modified L(N) where N is measured in compressed bytes, not raw parameters. +
++ This means techniques that improve the effective parameter count (weight sharing, quantization-aware training, better compression) + shift the L(N) curve leftward: getting the same loss with fewer bytes. +
++ At fixed parameter count, deeper models (more layers) generally outperform wider models (larger hidden dim). + This is because depth allows iterative refinement of representations, while width provides + more capacity per layer but with diminishing returns. +
++ The Parameter Golf leaderboard confirms this: submissions evolved from 9 layers (baseline) to 11 layers, + even though adding layers costs parameters. The efficiency gain from deeper processing outweighs the parameter cost. +
++ The extreme case: depth recurrence (weight sharing) takes this to its logical conclusion. + If depth is efficient, run the same layers multiple times and spend the freed parameters elsewhere. +
++ While Parameter Golf is framed as L(N), the 10-minute 8xH100 cap introduces a + practical compute constraint. At ~87ms/step (SOTA), you get ~6900 steps. This means: +
+
+ Train the baseline at different configs (e.g., 3L/256d, 5L/384d, 7L/448d, 9L/512d).
+ Record final val_loss for each. Plot on a log-log scale (log N vs log L).
+ Fit $L = a \cdot N^{-b}$ using least squares on the log-log data.
+ Use scipy.optimize.curve_fit or manual linear regression on log values.
+
+ Calculate exact parameter counts for both configs (including attention, MLP, embeddings, norms). + Verify both fit under 16MB after int8+zlib quantization. + Train each for the same wall-clock time and compare final BPB. + The result should show that depth wins at this scale, confirming the leaderboard trend. +
+ ++ Read each submission's README. For each, identify: what was the primary source of BPB improvement? + Common axes: quantization (int6, GPTQ, QAT), architecture (XSA, SmearGate, BigramHash, MLP width), + training (Muon, EMA/SWA, longer warmdown), evaluation (sliding window, TTT). + Look for patterns: which axis has diminishing returns? Which is underexplored? +
+ ++ The heart of the challenge: architectural techniques that extract maximum performance from a fixed parameter budget. + From attention efficiency (GQA, Flash Attention) to parameter sharing (depth recurrence, MoE) + to specialized modules invented for this competition (BigramHash, XSA, SmearGate). +
++ Standard multi-head attention: each head has its own Q, K, V projections. + Multi-query attention (MQA): all heads share a single K and V projection. + This reduces KV parameters by a factor of $h$ (number of heads), saving significant parameters + in the attention block. +
++ Parameter savings at 512d, 8 heads: standard attention K/V = $2 \times 512 \times 512 = 524K$ params. + MQA K/V = $2 \times 64 \times 512 = 65K$ params. That's an 8x reduction in KV parameters. +
++ GQA uses $g$ KV head groups where $1 \leq g \leq h$. Each group of $h/g$ query heads + shares a single K/V head. The baseline uses $h=8, g=4$ (GQA-4), so every 2 query heads + share one KV head. This is the sweet spot: nearly the quality of full MHA at half the KV parameter cost. +
++ Standard attention materializes the $T \times T$ attention matrix in HBM, costing $O(T^2)$ memory. + Flash Attention tiles the computation: it loads blocks of Q, K, V into SRAM, computes partial + softmax outputs, and accumulates results without ever writing the full attention matrix. +
++ Result: exact attention (no approximation), $O(T)$ memory, and ~2-4x faster due to reduced HBM IO. + Flash Attention 3 adds Hopper-specific optimizations (warp specialization, TMA, FP8 accumulation). +
+
+ Key: when num_kv_heads < num_heads, repeat each KV head to match query heads using
+ repeat_interleave or reshape tricks. Pass enable_gqa=True to
+ F.scaled_dot_product_attention for the efficient path.
+
+ The Universal Transformer applies the same transformer block repeatedly instead + of having unique weights per layer. With $K$ shared blocks run $N$ times each, + you get $K \times N$ effective depth for the parameter cost of $K$ layers. +
++ Key challenges: +
++ This is our primary research direction for the competition. +
++ MoE replaces the dense MLP with $E$ expert MLPs and a gating network that routes each token + to its top-$k$ experts. Total parameters = $E \times$ MLP params, but active parameters per token = $k \times$ MLP params. + In Parameter Golf, MoE is tricky: all expert weights count toward the 16MB limit, but only $k$ are active. + The benefit is more total capacity per token, but the compression tax is steep. +
++ The baseline splits layers into encoder (first half) and decoder (second half). + Encoder outputs are stored and added to corresponding decoder layers via learned skip weights: +
+# In GPT.forward():
+for i in range(num_encoder_layers):
+ x = blocks[i](x, x0)
+ skips.append(x)
+for i in range(num_decoder_layers):
+ x = x + skip_weights[i] * skips.pop()
+ x = blocks[encoder_layers + i](x, x0)+ This gives the decoder access to both high-level (late encoder) and low-level (early encoder) features. +
+
+ Start from the baseline GPT class. Replace nn.ModuleList([Block() for _ in range(N)])
+ with nn.ModuleList([Block() for _ in range(K)]) and loop each block $N/K$ times.
+ Add a learned nn.Embedding(total_iterations, model_dim) that's added to the input
+ at each iteration. Compare loss against a standard 12-layer model with the same total parameter count.
+
+ With a 1024-token vocabulary, the embedding table is small (1024 x 512 = 524K params). + BigramHash adds n-gram context by hashing adjacent token pairs into a separate embedding table: +
+hash(t[i-1], t[i]) = (36313 * t[i] ^ 27191 * t[i-1]) % bigram_vocab_size+ The SOTA uses 3072 bigram entries at 112 dimensions, adding ~344K params. + The embeddings are zero-initialized so they don't disrupt early training. + A learned scale parameter controls their contribution. +
+gate = sigmoid(learned_gate) # per-dimension, init to ~0
+output = (1 - gate) * x + gate * x_prev+ Creates a learnable temporal smoothing between adjacent positions. + Initialized near zero (pass-through) so it doesn't disrupt initial training. + The model learns which dimensions benefit from position mixing. +
++ XSA removes each token's self-value projection from its attention output. + This prevents the model from simply reading its own embedding and forces it to + attend meaningfully to other tokens in the context: +
+# Subtract self-value projection
+v_norm = F.normalize(v, dim=-1)
+self_proj = (y * v_norm).sum(-1, keepdim=True) * v_norm
+y = y - self_proj+ Zero parameter cost. Applied to all 11 layers in the SOTA. +
++ The tokenizer determines the fundamental unit of prediction. In a parameter-constrained setting, + vocabulary size directly trades off against model capacity. Understanding this tradeoff is critical. +
++ BPE starts with individual bytes/characters and iteratively merges the most frequent pair + into a new token. After $V$ merges, you have a vocabulary of size $V + |\text{base}|$. + Each merge rule is deterministic: given the same text, the same tokenization results. + The baseline uses a 1024-token SentencePiece BPE vocabulary trained on FineWeb. +
++ $\text{BPB} = \frac{\text{bits\_per\_token}}{\text{bytes\_per\_token}} = \frac{\text{val\_loss} / \ln 2}{\text{bytes} / \text{tokens}}$ +
++ A 1024-token vocab has ~2.5 bytes/token. A 32K vocab might have ~4.5 bytes/token. + The larger vocab model has lower bits/token (more info per token) but also higher bytes/token. + BPB normalizes this, making comparison fair. +
++ Embedding table cost with tied embeddings: $V \times d$ params. + At $d = 512$: 1024 vocab = 524K params (3% of budget), 8192 vocab = 4.2M params (25% of budget). + Larger vocab captures more per token but leaves less room for the transformer body. + The SOTA uses 1024 + BigramHash(3072) as a compromise: tiny base vocab with learned bigram features. +
++ How you train matters as much as what you train. The Muon optimizer, distributed training with + Parallel Muon, and weight averaging (EMA/SWA) are critical components of the SOTA stack. +
++ Muon applies Newton-Schulz iteration to orthogonalize gradient matrices before applying them. + The iteration uses fixed coefficients $(a, b, c) = (3.4445, -4.7750, 2.0315)$: +
+X = G / ||G||
+for _ in range(5):
+ A = X @ X.T
+ B = b*A + c*(A@A)
+ X = a*X + B@X+ This converges to the orthogonal factor of $G$, effectively normalizing the gradient matrix. + Muon is used for all matrix-shaped parameters (attention, MLP weights). + Scalar/vector params and embeddings use Adam. +
++ Parameter Banking stores all attention/MLP weights as contiguous 3D tensors (e.g., qo_bank[2*L, d, d]). + This enables batched Newton-Schulz and overlapped communication: +
++ The bridge between training and the 16MB artifact limit. Post-training quantization (int8 to int6), + GPTQ with Hessian-informed rounding, quantization-aware training, and compression algorithms. + This unit is where bytes are won and lost. +
++ Per-tensor: one scale for the entire weight matrix. Cheapest metadata but worst accuracy. + Per-row: one scale per row (output channel). The baseline uses this for 2D tensors. + Per-channel: one scale per column (input channel). More metadata but better for asymmetric distributions. +
++ The SOTA uses per-row int6 for large matrices and per-tensor int8 for embeddings. + Scale metadata: fp16 per row = 2 bytes/row. For a 512x512 matrix: 1024 bytes of scale data. +
++ GPTQ collects Hessians $H = X^TX$ from calibration data, then quantizes column by column. + For each column, the quantization error is redistributed to unquantized columns using the inverse Hessian. + Columns with large $H_{ii}^{-1}$ (important columns) get better precision. + Block-wise processing (128 columns at a time) makes it efficient. +
++ The SOTA generates 64 sequences of 2048 tokens at temperature 0.8 as GPTQ calibration data. + No validation or training data is accessed during quantization. This is legal under the rules + and provides representative activations for Hessian collection. +
++ How you evaluate matters. Sliding window evaluation, test-time training, and long-context extrapolation + can improve BPB without changing the model at all. The rules allow 10 minutes for evaluation, a separate budget from training. +
++ Instead of evaluating in non-overlapping seq_len chunks, slide the window by stride < seq_len. + Each token gets scored with the maximum available context (up to seq_len preceding tokens). + Only score the "new" tokens in each window to avoid double-counting. Stride=64 is common. + Tradeoff: smaller stride = more passes = better BPB but more eval compute. +
++ Legal TTT in Parameter Golf: you can only train on tokens you've already scored. + Score-first approach: evaluate a chunk, record losses, then fine-tune (e.g., LoRA rank-4) + on those same tokens. The next chunk benefits from the adapted model. + The SOTA dropped TTT because it was neutral on their stack, but it showed promise in earlier submissions. +
++ Every millisecond per step matters: faster steps = more training in 600 seconds = lower loss. + GPU architecture, kernel fusion, memory optimization, and profiling are the systems-level skills + that turn a good algorithm into a competitive submission. +
++ H100 SXM: 80GB HBM3 at 3.35 TB/s, 132 SMs, 989 TFLOPS BF16 tensor core. + Key Hopper features for Parameter Golf: +
++ Everything comes together. Read the SOTA line by line, identify remaining headroom, + design experiments, validate with statistical rigor, and prepare a submission. +
++ Where is the most remaining headroom? +
++ The submission process requires: +
++ Starting from the current SOTA codebase, implement one novel improvement. This is the culmination + of everything in the curriculum. +
+ +