diff --git a/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/README.md b/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/README.md new file mode 100644 index 0000000000..72bd33ea6c --- /dev/null +++ b/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/README.md @@ -0,0 +1,108 @@ +# Record: AttnOutGate + SmearGate + Softcap 15 — val_bpb 1.07750 (3-seed mean) + +**val_bpb: 1.07750** (3-seed mean, std 0.0006) | **~15.99 MB** | 8×H100 SXM + +Beats current SOTA (PR #1493, 1.0810) by **0.00350 BPB** with std 0.0006 → t-statistic ≈ 5.5, p < 0.001 across 3 seeds. Comparable in magnitude to recent record gaps on the leaderboard (e.g., #2→#1 was 0.0012, #3→#2 was 0.0006). + +Three additive zero-cost modifications, all fully precedented and reproducible. + +## Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128) + +| Seed | Steps | Pre-quant BPB | Quantized BPB | Sliding BPB | TTT BPB | Artifact | +|------|-------|---------------|---------------|-------------|---------|----------| +| 1337 | 4457 | 1.08396 | 1.09737 | 1.07817 | **1.07693** | 15,994,840 | +| 42 | 4459 | 1.08491 | 1.09617 | 1.07934 | **1.07805** | 15,996,097 | +| 2025 | 4450 | 1.08458 | 1.09555 | 1.07873 | **1.07753** | 15,992,597 | +| **Mean** | **4455** | **1.08449** | **1.09636** | **1.07875** | **1.07750** | **15,994,511** | + +## Key Changes vs Our Previous Submission (PR #1876, 1.08008 BPB) + +Three additive zero-cost modifications: + +### 1. AttnOutGate (PR #1667/#1693) +Per-head data-dependent gate on SDPA output, before `out_proj`: +``` +out = W_o @ ( SDPA(x) ⊙ 2σ(W_g · x[:, :12]) ) +``` +- `W_g`: (12 × 8) per layer, zero-init → 2σ(0) = 1 (transparent at init) +- 8 heads × 12 width × 11 layers = **1,056 extra params** (~1KB at fp16) +- Lets each head dynamically suppress noise per-token +- Routes through scalar AdamW (added `attn_gate` to CONTROL_TENSOR_NAME_PATTERNS) + +### 2. SmearGate (PR #1667 + PR #1851 BOS-fix) +Forward-1-token residual mixer at embedding lane: +``` +x_t ← x_t + λ · σ(W · x_t[:12]) · x_{t-1} (for t ≥ 1, identity at t=0) +``` +- `W`: (12 × 1) and `λ`: scalar — both zero-init +- **Total: 13 extra params** (~26 bytes) +- BOS-fix prevents cross-document leakage during packed training: gate is masked to 0 where `input_ids == BOS_TOKEN_ID` (default 1) +- Routes through scalar AdamW + +### 3. Lower logit softcap 30 → 15 (Modded-NanoGPT record #18) +Single hyperparameter change: +``` +logits = 15 * tanh(logits / 15) (was 30 * tanh(logits / 30)) +``` +- Tighter cap engages tanh's saturating region +- Smoother loss landscape, prevents extreme overconfidence +- **Single-line change, no params** + +## Architecture (unchanged from previous submission) + +- SP8192 BPE tokenizer +- 11 layers, dim=512, 8 heads, 4 KV heads (GQA) +- Depth recurrence: layers 3-5 looped 3× (17 virtual layers), enabled at 35% +- XSA on all 11 layers, parallel residuals from layer 7+ +- U-Net skip connections with learnable gates +- Tied embeddings, MLP 4× LeakyReLU(0.5)² +- Coprime-stride multi-shard data loader + +## Training (unchanged) +- Muon optimizer (5-step NS) for matrices, AdamW for embeds/scalars +- EMA decay 0.9965, 72% warmdown, 20-step warmup + 20-step loop warmup +- Gradient clipping 0.3 +- Brotli-11 compression + byte shuffling +- Score-first TTT (SGD, momentum 0.9, LR 0.005, 3 epochs, 32K chunks) +- Full Hessian GPTQ with Cholesky error compensation + actorder +- LZMA code compression (53KB → 19KB) + +## What We Tried That Did Not Help + +| Technique | Result | Why it failed | +|---|---|---| +| LoRA on recurrence (rank 2/4) | Worse | 10% step loss, artifact over 16MB | +| MTP (Multi-Token Prediction) | Worse | 10.5% step loss, no quality gain | +| QAT weight-snapping during warmdown | Catastrophic | Disrupted Muon's update dynamics | +| Hessian-Aware SDClip (PR #1412) | No change | Per-row Hessian importance too noisy | +| Per-group clip allocation | No change | Group traces are stable but didn't translate | +| Asymmetric sigmoid logit rescale | Worse (+0.001) | Tanh form was already well-tuned | +| nGPT normalization | Excluded after research | Speedup only at 0.5B+ params and 200k+ steps | +| GatedDeltaNet/linear attention | Excluded after research | All "frontier" PRs had byte-accounting bugs | +| Value embeddings | Excluded | Don't fit in 5KB artifact headroom | + +## Compliance (Issue #1017 conditions) + +### Condition 1 (Strict Causal Dependence) +Causal attention via `flash_attn_func(causal=True)`. AttnOutGate uses position-local input `x_t[:12]` (no leakage). SmearGate is strictly backward-looking (`x_{t-1}`), with BOS-mask preventing cross-document leakage. TTT only incorporates tokens from already-scored chunks. + +### Condition 2 (Full Normalized Distribution) +`F.cross_entropy` over full vocab_size logits. Softcap is monotonic (does not mask). + +### Condition 3 (Score-Before-Update) +Each TTT chunk scored under `torch.no_grad()` BEFORE any training on it. Model weights at scoring reflect only prior chunks. + +### Condition 4 (Single Left-to-Right Pass) +Single `for ci in range(num_chunks)` loop. Each token scored exactly once. + +## Credits +- Current SOTA base (PR #1493): @bigbag +- AttnOutGate: @MarioPaerle (PR #1667), @dexhunter (PR #1693) +- SmearGate + BOS-fix: @KoszarskyB / @classiclarryd (modded-nanogpt), @cocohearts + @aquariouseworkman (PR #1851) +- Logit softcap 15: @KoszarskyB (Modded-NanoGPT record #18) +- SP8192 + GPTQ + SDClip: @clarkkev (PR #1394) +- Depth recurrence: @dexhunter (PR #1331, #1437) +- Parallel residuals: @Robby955 (PR #1412), @msisovic (PR #1204) +- Score-first TTT: @abaybektursun (PR #549), @Christopher-Lee-McClendon (PR #461) +- Coprime-stride loader: PR #726 style +- LZMA code compression: PR #1394 diff --git a/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/submission.json b/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/submission.json new file mode 100644 index 0000000000..944ac415b0 --- /dev/null +++ b/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/submission.json @@ -0,0 +1,42 @@ +{ + "author": "Meirzhan Saparov", + "github_id": "Meirzhan05", + "name": "AttnOutGate + SmearGate + Softcap 15", + "date": "2026-04-25", + "track": "10min_16mb", + "val_bpb": 1.07750, + "val_bpb_std": 0.00060, + "seeds": [1337, 42, 2025], + "seed_results": { + "1337": {"val_bpb": 1.07693, "artifact_bytes": 15994840}, + "42": {"val_bpb": 1.07805, "artifact_bytes": 15996097}, + "2025": {"val_bpb": 1.07753, "artifact_bytes": 15992597} + }, + "hardware": "8xH100 80GB SXM", + "pytorch_version": "2.9.1+cu128", + "record": true, + "technique_summary": "SP8192 + 11L + Depth Recurrence (L3-5) + Parallel Residuals (L7+) + XSA-all + Coprime-Stride Loader + Full Hessian GPTQ with Cholesky Fallback + EMA 0.9965 + Score-First TTT (SGD 3ep) + Brotli-11 + LZMA Code + AttnOutGate (PR #1667/#1693) + SmearGate with BOS-fix (PR #1667/#1851) + Logit Softcap 15 (Modded-NanoGPT #18)", + "compliance": { + "train_under_600s": true, + "artifact_under_16mb": true, + "eval_under_600s": true, + "no_slot": true, + "no_pre_quant_ttt": true, + "no_etlb": true, + "no_ngram_cache": true, + "score_first_ttt": true, + "three_seeds": true + }, + "attribution": { + "current_sota_base": "@bigbag (PR #1493)", + "attn_out_gate": "@MarioPaerle (PR #1667), @dexhunter (PR #1693)", + "smear_gate_bos_fix": "@KoszarskyB / @classiclarryd (modded-nanogpt), @cocohearts + @aquariouseworkman (PR #1851)", + "logit_softcap_15": "@KoszarskyB (Modded-NanoGPT record #18)", + "sp8192_gptq_sdclip": "@clarkkev (PR #1394)", + "depth_recurrence": "@dexhunter (PR #1331, #1437)", + "parallel_residuals": "@Robby955 (PR #1412), @msisovic (PR #1204)", + "legal_ttt_framework": "@abaybektursun (PR #549), @Christopher-Lee-McClendon (PR #461)", + "coprime_stride_loader": "PR #726 style", + "lzma_code_compression": "PR #1394" + } +} diff --git a/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/train_gpt.py b/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/train_gpt.py new file mode 100644 index 0000000000..bba44a8a94 --- /dev/null +++ b/records/track_10min_16mb/2026-04-25_AttnOutGate_SmearGate_Softcap15/train_gpt.py @@ -0,0 +1,2 @@ +import lzma as L,base64 as B +exec(L.decompress(B.b85decode(";Mm|h4_yE>n@VT6Qap3bt~@<3h>ok~)Km^%c^ys%R{D_%yAk9-_tV7^coUOo3$w>`(`ci)t`2F7>r>Ltx>>S2CRw|7ov>Wn1e~_!RLQ=%V9g?)G3yPsu%SBy!lj1PaC-x%dDmCDOZ^r^!)+WWz}ejKXTJ#^U6Ra!};QocHHXQC+4UM!QQ!-N5Xd|%~a(9)bTYIO+>B~8~@lqmri%^qEkQUy074Rh6w7V_#^s9J-3BNA`G;qyR$LYcI?e+loZVWi~B$n=TKFp{%SeHYp{oNWh;U@Ahk8M2$OU%K8B$lb*dRQXd-GR_@*KAZdRdwSd#v=LSq1v@Puul=a7WXDmh1^kBj}Y2XlER!D2E{&{%lV(hz$#n5%+%sk&Q}>{y0xpRgiQQBJeVV0hy8UD3ntyo@(Pv+K7^zVRDt4bah(r8kfsZThb+H1)~K-lIr4`|V#-2R>G7pP*N!fwWd&Dq8C)y=NrG_U_Oz6Q?+@ok1?(VJ5?ZT~&}C4Ks38WRB>3i=I!}H-8qq=&yKJ;tbpwwn~lAseD^q1C*u5T;lKQtF;?zv@u0f36%6SXU~txi3v5iSPK*`fNE9531KaQDL`zTPF$MX4U(-3sY-&?>QJe)giBQzpor7H)AZ#4=Hn#`AoAL7tT){&bw+#clo79(exRGw6=3qI8d?ENaN|9r<$Kv?)KMQ&PP{<7^riC5(v?@Vn$#!?ve|=((71`tK0b$pTCJ%7K`fsaMgFa#Xp&Cpz%dRL=LQ2^d6*22a=uAcN<+vQBzf~liGB3CgXDs5DzO<~Q>p5Puc3aIg9g0DAk@Ks(-8TPQq~v%`i+cmvAYD>kh>zd&m%xhfe%g4KTrXozWp-;diPVx5JCV|_*_Bdg#`ZJ%Zm7po*mx1j%nk5N>lXOp})`cOZrW*L@^)R7WL66!#p5Go`+t?_pHu!if>=hBR%o&CK^qGbo43`g-l3{fi8)fWi&hJpP#Rli@z-zA0FxOZOSTOn!lu(k8DUHN#d+z2egT|1tKI}CE8{(X+{4h(1WWi7Q_rH&tVhMZQI!{QGrJ*q^l)VxV+-N}yLG~{to5Qs64o;iBC<4^k)_^*^z>=Le1Nt-8OK^IIfD)2GmKirUSO!H}>}%H}rKL9%fl<>#5f}ez;R;IkDQIU?em}<{ybqH+ss#Hrx{p1>{69ls-yuE#)wL5Uy!kOLw@d<5u{$j~13ITSwJhDwr+Bt61PkSsjFCAx)d!sQqtQZKZA6D=rX9v`?|nodaT_JpoRm@Vl3Pr2t^g|f5+-*CwMZJ$-7lLFF5l(kRc0eG9S>F#CPIE7y_PJMUhxnY5>@uP4YW$Wc*x<7hr5oPkK-dI<<9PySx5iJN1^+6mb(-~=*=k%Knj@={EGJGsc;JpZ^3BVDE#z&b1sP(I|1n6&x)uQS%edDva&b{<~n~KN~w8%Xvk(eNrCLiteW`Go#`eCo&IY4Sz%-{@b116!QAI7l`2+dLRozb`rd#Bd@|5!(g>|z)@z=c(G8C2;d51<4!Y|ihrO+$cbeQdGdZ?=Zl&d@PA_ZpOXRBlRJ9U{d_&RR1;d2`VE$UHD}v}?tMr8l=`m)AVj_AAbKI)SvvJiPqlHf@tGsI;OKdFv(@W+sdDIXbcM#1j7rGx!UnqFQIoaOn4!$XrygF?=SQ%>edK&~)gH5;>TD)}YuhGXV_82^GCd+yNG*aG^S>~JUxGgcM{A?{6X|>_`MIPeseS6J!%)x%cr!Y!%#s6&N3>9Nxaw~AEB0}q;K1MoWl}+$vg@!?|bUffYoF}y-*INfaY4ufTs&)>il$k^T;U)eU*~hsOfaSb*8J;yTY$!I3@**!3!Z-@$rF3!VF*I%P2{-ZO5N!Q1!E^bRcwMxfbE2#bLqL3m&3P4sk!r5LL{q4a8m-pFMu88Q-P1|rPpUqva}VOhb6gDzG@ndCs8OBu=|Xx;mq%e(91S9*!O>-)O;%iqJ?u|{zokJZWkTl*lzQ}YKE?`Ud*fdBb@XS>k$y>(r0L&x{qnjT^Hi9_A-^sc!X0a2T@n9+_O7%q71Q>fP^%}WtSVDSzWFX=@CKI@VRIx|bpGl!^K>Bl?{~$gb=q}7kh3ZeQ5^)D)ASN<_GbXg>>K>K$2Q>Ur~DAOjX5-(^*ld#NMVsnq0R{-LZ3p19ok1_-KT7IXpVR`j8JPEivHI10ecMGTN`dy$KD?xv6&}6{n+PV>gCVq^!vZO^B?zQ(IF~a~=xGJ6ITa4NE#Z;C2`aO}je>B?`)!#^_tKraHz36G4aJa%@rCP9-s*A9*KM|W1{N^!6kK?hgkAcVmPyllOQy)39yJhF86dOK!LcAq=*sHpj?PdV!PSw#B+yjPO?nnM`+xQ8(t_4T&(2FqFM-2(xx}eIb4+dic}=rTEWY@CyO*(UwkSztFYlY1-$0f7GfgpW>KTcb^LC;+|h7@Gg2`pre9w0PDhn;o1J5IGLz%UR1s^GAa3sJVO;!rAc8tJU=f;>XR#ksuUUS?={&M!(e@y3rrEw4OncDy_8epRxfD*&61JuA`87)|SQw&ylZsAt=Ch!+gRqbV0+uIox;dhe!)AO;Q~QLjfbGU1+q3NC-E8idKGP!E+;Pl*25u{&V@Yr*>E6+gY}kG{P?`krmD8C5CpQ?`Ar9o9mH@xYVBs=PX{@$dc`UzFCCU|z5i&8FSlM9?(7-8=@`Hz&=UE$>?HVtm*+-|{EwoS`S^0AYD;t5E(ayMX0%(bFwGt7u8MEz1NMun7Mt;Br_e@w^Qq}7o;jB*8xai+z#ON|+o0hOW1=;V>;zXYQulo5Zw{yCS@x*e;aP2Xq^)0Xx#NtyXW2XC87oVWwcGyK>@r3`M?k@?BXT&po~qw1M)%DekiNfX+fU>d#>2te5JO`jg2opyp*0Z`nvB^;GgMpkpSXwc`bAp4vnU(*jAPm|!fvej~6!;#2OKR)|+jWF94OsW0;9J|S&<>J!tgi=Ewna+VjVzaQK0e#8`KZDMxjgcq#e73B<$U1&5CK~I-K1BXgPzqGhwpO8g+i;%mh`kVbYTrv|g`+c<3>AyyTLH5jcd*R=NauKSycP#m?(9;p`vca(e2LNsdsziI5E}VOZ9R~_AX-uXexj_udN1OY|cr?H-yo_cc=mT{X;*rNM|BwBA^ri)&KJbbx^Be;~V*-?#G?nRIi)!r8F-zM;`nYf=-pP?aINh&pNT&LCzLR%lf^P6sI~fd4q29Z($XdD1L-#0R=$sQq=LY^|_aTL){1cr|wHk9`9G4~j7^H6Kpf-`fN@=y@`6iy4L;-&$EGPBZ3drtGXeE`HV1RkBTaV32k@KQv-iGk%2iL4G^Yp@puMbG_Z|7l;mAT;FueFhg3rIX`meCMb(y-$%6z01(1aXcnr$Kym=k&U|ejYt(1cG}KF&_X)Pgn3sJ(gCY-D_<4p(ctO)CQc0!+^xg8eoIMr?xcxw%0n3xgUBc@)m3?Byq#c~9Oz1;W7KO4oAk4n^k5X#1LZ2RZdbOeh3G#%uQ$U{TLL$A(^Je@m4;*l3=6+wT+%6J_5F8N!jk#-UI5o01($w*N`I`%mGCKqUm@yx{IELnRm?MGwCPA%J+VzYxqi#kGZIXYPsUkHJf}z1m@}ue;@ia5%$tL9kpU5F@(BW!z8H{N6o(q=#z6L;PY4`$#dy4KtEi7d|?!RNIKh&5!0`Y}#7|zlKKY=pDelL4!*#Qm?z;7qr0EJi>ElL8SnI`3pPIO62D95`gAvXfYwV+Srp1+5T73-|N-}0{pvT`|f4@H1~*MxM0AL%SYL2RaZX;4pb&nuBfYa%bP`=elQ_3O>`5edivK?pV&cdu-+;QGw_=`kGJ${2re~)qMH8xhTt?kt{!YfY1=!;Mybns34utZt5_pq(Ais-d)|3a&lj|qw9(Ubk+QeOK0Vj;*QO^uji?)6@U5<=8S;v9Rd=Qr8~_VU{5`Q(G-A-$XRogH%O>VvPom=;KTIzw#CH0ddi_LMs3P)V|zR^^U|=6dDdmSrH{J$x61%twd}u4;xxgh>6Q25m0Woh3@=Cab8E%*Pnr@bVgcg$uzP4C^zfVdgSM+qGT0y_2)$zhwzm1ss^(G#?z$X+X@M35XB3tC*IRM1R1u$a6vhQLR#<=nlW_l1+APe(o8Z+6?VV-gH(foV?tK%^`-b_a=x+k6Rd%O?}1mWozP7ddB0Wv+@69>Zd?91)4B`I*ac`L^fJ>;@ODHv>vXIw(I*vC?eka${g%RyR${{AJfDg`X!zZ=z)l-$I7KI9=CmkhiV$PAdYW$gdip7S(K8S7G<^yrIF&TDmjfcni>`~@o3Anx@04)aVdfm~atu6Ei@b_g_Vfs}7c72pY(Q|^2I70jvW}St(W;F)z4Aj_Byz6O{a>1)S-+00Hb}qaK33;C;0MQrtbjeIWiA+N6zoYj+W1xY(&*${l&7`w0rQ6u4?<=y?Sm_VN7PX#IW%J2h`-1pX+(yr6?R9ThW~TsRB-r{l}_=DG8!thlz+$<`HP$P1xanDe+OnU5F>LgfX|a_t@chWR`I*-dd~yajDbzrb$!u>A!+kr?I{EO`xhB*)6<`u|BD1=}hh~rY)vn-HaPmvHQ@stYspuP4I{98)klv;iX1q>a(j&m^E*WmE>`(4!%o_p5T{XxKO;UdqQ64L+PAjHKX=Y3Mfq?KEztZV~q@SzmJzE;3$Aqb^ntGs5`qX$^gVJtO{L%0wDU}FmdxcUdV_t{$H#>3uOd&y~!pbU<3TO%A)oeIQ6c3sNkM~)^7^s(%QtED8AS;9ZCkr5VrjU-u@V;%O|wKV0YZ_ZOyanR^2-7Qf|HU~gKcA_>mdoN;61$8;68T11bovTx&A$w#pR3m8BI<;gM)({6ZNpyFMQvlmX2Kx9r2_gUM%dnQ@Z0*<(cg)P;F(lD8c^{ubXs%6IyfQ$nB*8L;z~Q-pqZEY`thp#-=?=H~HJ8Jn)|JI(qzxX`PC9{rPRB^FogFESjVqWL`#@-&HTq(M{Ompwa2k_k=cETlcyQW#yMtkCa2433d?3aN=Yc#`)4=nLuXZitdPYqv$RW?wkW|9(VXkfsvUan2k$&EPR3`g>(#l>xe`9#Ht=Vx(Ad2VQTBuAS03}&-A`H4`7W%N_GgM769FLaY7U;4;U~knmcy_T#hz8hJB+rM2!UBjOVSn70fn9YWuvdeJB9=8t4RFs3B{^du1NXCmFiWtrg}Rd5FNxenFtS8(9=8tfl&})$r9b0o98x(ua8#`15&1wwt=n@SOV>YtflX%4k6JUvJw0*&vG%)ch-I6i8=?p&2J-^ZzC;(W|a@C7K|Sa>!8Wl}I5~Ls&Zdky9pEs*Y6}FjD+@x=-q%r2+s2DNsedlYjm4p`yAWUNNFyET5%^g!|<88^EYVKT|lsinq0P(tA|Azx`J$+~?8=m3Nq(8&qOUx#-XP8+B_`v^;_ABpgSf#dew!tMv?m)0a)rAg}PqOw3zCzK1<$2++GZ&TP5y5i)aNc}J+>Z3V1(1!L`ZG=XAcXJfyY)%UvcARKUy8u8X5?4o6L`m#*!NseMiECX%Ih?)Tt}Dcbf(_{#g+8cyG$7DKBCtww?H&P%hzk~(Cl-UqoLYT|0`#DtjuzyXr5eQJfOYS40QaxYr;iIwm7S8jiBP)zs_801D&2wd+NUsfQDYZSmL>D=L7(XxY2cxZ=#}WEYIDg(Q?d1w%qMOS0#zaDxxX#Kvs}S+P;XyP3{OTr@(!@swjs)>}PkGR=~mipCFGrP%{af2Jqtd*`!{t#>*D;kXXg5P%ciZ5nP|%J0I1$Kp=b`#mU?a3(}iE#Za4R4;!AGxcm%9X6((Ld33cj)xwj(fIi6R@mWuN?p%>H_3RO;RU|XIB@;-Lf{2FiDPX3;T$8)vpjKi@|mz87gWc(W3;TS+;ItGyQm=+d&mw69@%ZZ^l9P@}Somtz-(4Z$oDe-XP`|Uvb8C8ooo8YOh+)`wKZ5O`Aiqo=PMo@8K`!0}3}in~zF(A)~J#hM2Z9)4UAsQ^};lIWdLlB{Gs8e~XKS>hNmivg6+923=M=@SazdsxsuQm!u5tJ#~9ii7@Feb@al^g(7RIlW~k#Yi4y-+5zPfS;16j4j5?;Aqtb#RMo-P&onq?lBTi)Hay~SuG05G14;z9asGzNmrxdvJTKU&%UxaSl$O4b%6l&QXSS5ShLw=0CLTnhR9|)_K2`BqS%(Zg3hp99Vq}4}W=Fs7ANpBinfjXl2fI}<=?w>=i0zE#QXi#zM}~64mG^yE^OHRDvd7TQfX|ZwCux~9)1cPc)VnqNYMm6y*?hbMAf7gQY9`A5($N7IUnXUAPLP!SVyq(C>s_5%6zdcbXPKl34C0&$5zDmx)Y|TRQ`orgLwtXh1l*)@L!N~mhv@jUYzru|`&X@3S-wPKsxeSL(ftjq<0>`lcL^%H&LStyOExhtVkX5$qjp?Q9}KS0sDk6X0O$x5u)<9jPAy&Os-}$^uUq#4jaoG-&kfGP0aN~#@#t*|hM4VQSFod4V_1gJ^$AdOjl#c*)ADEm01&caLKIJ-35le$A8aJ*Y6epBbmYz8l|if)Pj6!yw}{O09jr7tfZ-z>8flqd$EBI;X_{K>oyZFa8YD6jz@!&4X$K3+Kd%Za>zFgYDSo;YX<(IS6)5$h0o)&>|JP(SKAR|VLYEldHnoz2>YP_v9fEYCgJ3>yEuq`iMy52Vx&+X{$A}TOo-3b_M0H3YSF3AYZ24g8|%fxDqp`%Nfx&MfTfaz6u}*%&7VE)iDz^#0G)b&Au5d)i`;`Xwj)5+tJD7UeQ$Z_4WHPu&S@7EOar+_{sQUG50f6SEE~P=g4Fgpi1ovXjL(*SyI;p;YUrK0BLM&?@MJ7#=Vyka2h>*v`b9|lhTg96xy`qEWo#R}_&pu{vt9pnvs(CS$~k{e(FJB$@w@0K07S8R#88a2rGG{Gck)S>b#Hq;+o7y-5f&Ps&eb9m8CR2D*ajKCZ$lYfNPdiuN6BQ)uS(Yih3Q3$t_A_W9G8eHAffcb3MBLZ2=A1f`QGn{f}VXt%;ca!eHd{00!mrTZ_y1PAHrf}_Jthz<8JfPc?I=|pkRze@UWP3C?3cxH$=-!$UMvNIH(Fa((b5nGI+3Qd{JC9V5^N1&1b}nemy^<1Tm`wxkNjgbDY|4>t4X-g_uu8cj&s_$4>;B3gos1p8KPM$*Bip%vp`KuiT8D7YE6W^kT!j)7QA!#!`~-kfG_{XDc+EyXG_IN0`IH_AJ+V7{01=U2pi{^U+$RR#SY%!RTioncsSXqY_s1OOAsN*~V^+Kgkza7+hoik%UDqi7%8%mpL~o-P8^ddS`SAM3yV-dl)X#Oh4-&J?6>ZT*{v&HB%jvNp*e20g!TRPC{1C}YHt739OvSJ5Z!@!HpSU(f=a#sh@%Ywwc$9KtZh4>vvYIt!)%NcKMMv|J7J5AaS3kHV6PrSKY?5jQN+u_KY9r3`78>HS(i{V4|U7+j};7{v!MP#2iav*4FhBYVOCBQD^lV14<>vWljUi8L+Cj4aFKg(Eli$#1%uSP3q(;n$9Lh7{WW=RW3*1=Q*9vqcDyYAN=2`nH}MKnfvhAEP;65O6R&z%w(B6>TwbU~y6%&O<#v;tj<>r?Jn%eSOxPUJ((=w5H)z9rW8k_ne@6!G#PR;P2M{gwT!_ezO`wJ1fO{?88)JRRD)x>27`^TWr27s2-CJ-Yu@`fs?#v}->u-L_8uq;1+jrrtsSd{VlG<8q>VKjbtw(D!HOvmnSa(GAzf0|IsP`E$kuplAB;x4o!{9~e$#QzB>#aDz0IOEAaE!q6l<4CX~-o+T25a)XT8NuBLDp!gY(p%l|2C{l)H|^SgvI3j2X;0pekSJ-WJ~bwiL$yOUCLuD_OMrQz(+h6AZN%POh0+p^sVb*e9W+&|8H^U8xt+`Yo^O$rsPy8+>VO1wHS0AKGt!D|3H)xgwH!At|@~|ygW+1WgWJ7MmW>kZ;#AD=uNBTF0=a*{klzq>`9f@?J8*B1`o0m%~gduRGq6rrWs0(cZU+NkG&-}1f^BPW%Nb5D6-39@@9;#(->??_5sX~De3`t_A+)&(?2&3`r2M%lKCRp7m3#LWXU=&FN@O7+R=sBhr-QBTy)-{-gp^?$_gSu-T3qq30s{abYNP~dKN!f)i#V+EJ@9z7gO?$HU_oIjYHwc6Fo9O?bI^`Ub9BxW=@buurEMS+{fIAc-;?rVw3VtRSX_$oW)LIn$-U+xb-^TS?_D1rgQhD&aB?TxiL8Ez#N8%~?E;*~)sJnQq;q?R%GjK^=mHN>z-kDoFE(yME?wqx;pmlkv+dBTNmBahV9YHW-@@T_51lQ2q%q3tWSBlZn=gkKv+p4#E%Bx6iCrICxFup=6aslEx7bMv<+p2BRmh2K(fdAe0+w(-9sZxwf4ol!(ICQM6i=_5r)rw#&6e~3e)D>SPXkPmsStXh!+fbO0xlHe%*83QHUPU8xjmPXS;8|7(QhJ7wbT}pw4FxrpuoVq)E7vl0B8>brM1&m_TNQi%WV8Nux_NIvL98SuHh2Y4s@h_dgGG@(r$_<-@me%LO``8Mg6wd|3(?yuyrJI*drW)OKgX8_N2fvi6GAhBc}&j~n_^ffNC@nYv9Xomv7>i=Xk;LtFv0LtH(CQZ>IF0(*%0r~O`g&yzWSMhAx4h;Y>#N_Oape;M|=>u*OPN~h7ZoaLd>A-L{Lj?fU!^3w2}MYQyp}4Yo*2ZOu=%p?o=Ibtcq2m<@dr7K-x`({50b)ZmB|7MK7l#v%c&jI|Cjgah*k;KuLp)N94T*-g)rYnBgeg&M8v-ALY~rLS~!g|fkw-xm6Ri<`{Wk9Ad#Ua6DeA!sv5Bom9hrjuo_3G*ow-IP2@i=vlhj+5~f@~)!N(*Bll`SP+!+kDMCXa)eYEr6q$uygnyl0F;(abM^FgM;!MNI(?H896fKweS&n|ZcS6GgA#HyLt{YYOgCg=)Q%>Mk+}rS*SeDE$VdS70F}dk_&t8#1?yHzT8FEd|t&8-&AbWJM90#y1rl{fTrMi*8q9xL!lOJa%~5Ew=jS<;En^wv*~TY&4_vrWgCc#mpqS-_e{`)=0`3l!3n(Z|VXBcIYv{1C0A6aVV;3u@e5Fnym0Zk9rWE&?5DX9OI%qt&2vCc6lt5A*0u-c8O-1Gpt@A{JCBJfzf95w18Z;6S%qddcWAYQ4E`fZ{BIIh43CDNTjP5f|%PpO*g?6jcM&XME!DmvYS!%USZs|-VcDwO^>S&K6i>qiac^F-m8F=Kd3C*rK~JXs0btbwKCBA2T!{y$;j0uV2Uv-Hi9+RzM2)Msw$RZY)~+TjeCIAz~lDT1n1w?3sx+KR{sz>roU54b)%qnFVK(_p6pGmkY2zWEYMj8!zu<+n$#nruP3v&unqRLAj{XxsQ26Mqk5g9V58MP?>S~Rk(+DFKic{J2efU#Qs{fu!6KKgPi#5Dc`PM0;j#HqL^5QwytGD%D@@fUvhr`%!xzXI%7vv=@x_HH^-MBB3t~L(d@SAN7k{g5$!ex~gvshn!$gO}mwjhEpYl#8>PY=vja-2)w*K@wQfFOslqny?}W0)@rngL(0GmabzRe~>$N@03`#55C|dVXTSeRM+bl!j;f8j0=bg%*Fg&P?%88|&ykvIT6glO$AR%tNx9g)ypDI{@hA4$=EC-4$K&3n6|7^eR>V2I^|WKb2ok8rNl|h9K0`QJ_sGr)|{EH{k(U^dsblgqQ3I$MG^A&nYHZUeg^jukEi`&lIC>RjMb6gECR~$2uady|G_;HY7cS_e^1}xB_x=7Yw@fj2eF4wchp`d@_WQ56me=a1_o8Ch5$=OwU<Mso3Dk7{`pA}$^aGgw2WC-yeIC-v$47FRNVnIm1df3v&N$CEQe(8Ci~@x^!Gz}?&}dY*9z1tA^>??%c=v{oC@loOu79t~_J7Ca82d0yp+u@;9@otLlK{-9K_8^EghD)ZEd?12s0ph^__x{5Daz82LGNCm&TV7&DLK?t8(hn(rK-DTE#{o=MP4(!k&soLjHdyZzkr2ud=3=hBBavqa(WGUV!{k@hcION(hRFEEK!zN|va#Xq)HkuvM4E_q?5$C+PU!+gY>LkYpz##?qP4N(_y8a(eNvUBhJp|)?PtaAna*(uCT>4pSQ;cMzzA%{v?idWto^0&lB}m_UW9s;)~~W$i{YEzMP69=ecU(c%p2oy4){SZw^NASv=j_ck4y%Jqqow(>h!s@`R;XRruv7>!$+}8s`+JQ5vzd?ZZ8xrgxe;8a;X5B0W{1}mL%|+Nki-WYj{G_Et6hMTePZjc&?*MyZ}#``#_;pmN^J>zzbF@|H<O~8Iax!mAj=iyF(dpYsKON8JxNg|rQ~&n>Kw)p+qXv@3qFt48#$X-6X86UVZ!~l%2(FJ|SgTrohx;mxc;|q)-<_q$?48UmWO?JfpnyP(8RO-Y2*?glLQjkIb1+S&W~w1%-#Sytve-r1nCtnMIlla!?*{)@>{`boRtT)v7v?3ardBzHTZc!+a*~NOw-51&|mZ2tQZvbnxB&f#o54qeRRXa+8(iOWH%uk$mxtfb{{VePXe))0(Y^Ms?UzjilFCOr>!u6PDRs$^PZyd9-1Jw?dOX7+E_~Vwmvcc%%atRL_RhW6LG%%Y}R~#@T?TmF^X8yn%11Nc=t8u0uM3--XHXpQZR&|xq)^@aT1}qo*?K-kOTcK&|Be>Gm*4PnVolFCu|?|)_1FHK?_ec_qqz{HGK%jaiTPQ2_759HZudWfJyc`#08x~RZxv*(DxtMMNuHdM&{LeTb4Ac0Y<>5Pj<8}3~;UavXXoUAX582iw<%uAtvCs4;EM4Kt=Pg!s~g<4O=R|p|_&}PEb?J_wG{IEcv7O~Ia$vmDIyjX)b!anfgV`)cj8sOQ-L`{L5#^wra_`B&Ul64&34}PbsIVfHPPsI~TY(sjoDFdazcALgfDydFDlJ_bC$Zo{cg;9mNmO)y@jj)1sPmq#S%4)L^>f_0hNz>G!t=~dilWr@hqC{FiYGW{*tVaYaJk+ePeJjnPywFy_sXsjNMbp2b&(4a|)f*I(Rv&tKNJT|sr(qaco5yj(-R+u0ed8$=mQDL{qYcw{2^PtBDf+y~TaC#hyXU_>Jj++1iwVkL>`QUXM!XAfsnvOxXHr_gEdZ3kLc}%h=ZKx#(cv_g3msC+yEkNf-v4ghWEan9Rtj`Q;B$Iw4D$pelS+~zPHg%)5);D6+Az9fKm<-(G=s`ilsI`e?Z_$H$r;dJOr^ec!QLLAO!LN>dgS5ir)$Lk{g;f)b$jykFG$-G7}imkH*v;BM|6;nU-@Ff&;S4"),format=L.FORMAT_RAW,filters=[{"id":L.FILTER_LZMA2}]))