Skip to content

Feature/gemma mtp - #3

Merged
Ooooze merged 7 commits into
feature/turboquant-kv-cachefrom
feature/gemma-mtp
May 7, 2026
Merged

Feature/gemma mtp#3
Ooooze merged 7 commits into
feature/turboquant-kv-cachefrom
feature/gemma-mtp

Conversation

@Ooooze

@Ooooze Ooooze commented May 7, 2026

Copy link
Copy Markdown

Overview

Additional information

Requirements

Ooooze added 7 commits May 6, 2026 20:58
@Ooooze
feat: add Gemma 4 MTP assistant support
72f60cf 
Introduces support for the Gemma 4 MTP assistant, allowing for enhanced speculative decoding. This includes new command-line options for specifying the MTP head and draft model, as well as updates to the model architecture and tensor handling. The assistant integrates with the target model, enabling efficient draft generation and improved performance in speculative tasks.

Changes include:
- New command-line options: `--mtp-head` and `--draft-block-size`.
- Updates to the model loading process to accommodate the MTP assistant.
- Enhancements in tensor management for MTP-specific operations.
- Documentation updates for usage examples and guidelines.

This feature aims to improve the overall functionality and efficiency of the model in handling complex tasks.
@Ooooze
feat: implement async MTP draft pipeline for improved speculative dec…
46fb4ee 
…oding

This commit introduces an asynchronous MTP draft pipeline, enhancing the speculative decoding process. Key changes include:

- Updated `draft_block_size` to 3, optimizing performance based on empirical results.
- Added new APIs: `llama_decode_mtp_async` and `llama_decode_mtp_wait` for non-blocking draft requests.
- Enhanced documentation to reflect the async pipeline's functionality and usage.
- Implemented tests to ensure parity between synchronous and asynchronous draft generation.

These improvements aim to increase throughput and efficiency in handling complex tasks within the model.
@Ooooze
feat: enhance speculative decoding with pipeline depth-2 and async MT…
7d2b887 
…P handling

This commit introduces significant improvements to the speculative decoding process by implementing a pipeline depth-2 mechanism that allows MTP draft computation to overlap with target verification. Key changes include:

- Added `prepare_next` and `cancel` hooks in the `common_speculative_state` interface for better async draft management.
- Implemented logic to drain any pending MTP requests before new iterations to prevent race conditions.
- Updated documentation to reflect the new pipeline depth-2 functionality and its implications for performance.
- Enhanced the `common_speculative` API with new functions for managing async MTP work.

These enhancements aim to improve throughput and efficiency in speculative decoding tasks, ensuring smoother operation during concurrent processing.
@Ooooze
feat: add MTP acceptance tracing and enhance draft processing
926ce1e 
This commit introduces an optional NDJSON tracer for MTP draft and accept events, controlled by the environment variable LLAMA_MTP_ACC_TRACE. Key changes include:

- Implementation of the `mtp_acc_tracer` class for tracing MTP events with configurable output options.
- Integration of tracing logic into the `common_speculative_state_mtp` structure, capturing relevant metrics during draft and acceptance processes.
- Updates to the MTP decoding functions to utilize in-graph argmax for improved performance and reduced data transfer overhead.
- Addition of a new shell script for running the Gemma 4 MTP server with enhanced configuration options.

These enhancements aim to provide better observability and performance in MTP operations, facilitating debugging and optimization of the speculative decoding process.
@Ooooze
feat: enhance MTP performance with in-graph argmax and acceptance tra…
f50b381 
…cing

This commit introduces an in-graph argmax for MTP draft processing, significantly improving throughput by reducing data transfer overhead. Key changes include:

- Implementation of `ggml_argmax` to publish final logits, allowing the host to read only the necessary token ID.
- Addition of a diagnostic feature for per-draft acceptance tracing, enabling detailed logging of MTP events for better observability.
- Documentation updates to reflect these enhancements and provide usage examples for the new tracing functionality.

These improvements aim to optimize MTP operations and facilitate debugging in the speculative decoding process.
@Ooooze
feat: enhance MTP tensor handling and introduce centroid routing
5d1ce14 
This commit improves the handling of tensors in the MTP process, specifically for the Gemma 4 assistant. Key changes include:

- Updated tensor conversion logic to maintain integer types for specific tensors, ensuring compatibility with centroid routing.
- Introduced handling for `mtp.centroids.weight` and `mtp.token_ordering.weight`, ensuring correct tensor shapes and types during processing.
- Enhanced documentation to clarify the new tensor structures and their implications for MTP operations.
- Added new scripts for quantizing and running the Gemma 4 Edge assistant with improved configuration options.

These enhancements aim to optimize the performance and accuracy of the MTP draft process, particularly when using ordered embeddings.
@Ooooze
feat: add TurboQuant KV cache and weight compression support
5cf0166 
This commit introduces TurboQuant, a new family of WHT-rotated low-bit quantization formats designed for KV cache and model weight compression. Key changes include:

- Added support for KV cache types (`turbo2`, `turbo3`, `turbo4`) with significant compression ratios.
- Introduced weight quantization formats (`TQ3_1S`, `TQ4_1S`) for efficient model size reduction.
- Enhanced documentation detailing usage, backend support, and practical examples for TurboQuant integration.
- Added new command-line options for enabling TurboQuant features in the server.

These enhancements aim to optimize memory usage and improve performance in bandwidth-bound scenarios, particularly on Apple Silicon and discrete GPUs.
@Ooooze
Ooooze merged commit 98bbdfe into feature/turboquant-kv-cache May 7, 2026
16 of 41 checks passed
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@dralves
common : Fix corrupted memory error on json grammar initialization (g…
80ae514 
…gml-org#16038)

Initalizing RESERVED_NAME in is_reserved_name() is not thread
safe and leads to corrupted memory when used from multiple threads
as can be seen in the asan trace below. This fixes the initialization
to make it thread-safe.

    #0 0x000100abd018 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void*>*>, bool> std::__1::__hash_table<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) __hash_table:1565
    AtomicBot-ai#1 0x000100ab0320 in SchemaConverter::visit(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) json-schema-to-grammar.cpp:802
    AtomicBot-ai#2 0x000100aafc48 in std::__1::__function::__func<build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&)::$_2, std::__1::allocator<build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&)::$_2>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&) function.h:319
    AtomicBot-ai#3 0x000100a2c938 in std::__1::__function::__func<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0::operator()(common_grammar_builder const&) const::'lambda'(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&), std::__1::allocator<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0::operator()(common_grammar_builder const&) const::'lambda'(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>, void (nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>::operator()(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&) function.h:319
    AtomicBot-ai#4 0x000100a139f8 in foreach_function(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&, std::__1::function<void (nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)> const&) chat.cpp:762
    AtomicBot-ai#5 0x000100a2a7f4 in std::__1::__function::__func<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0, std::__1::allocator<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0>, void (common_grammar_builder const&)>::operator()(common_grammar_builder const&) function.h:319
    AtomicBot-ai#6 0x000100aa98f4 in build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&) json-schema-to-grammar.cpp:982
    AtomicBot-ai#7 0x0001009c9314 in common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool) chat.cpp:1110
    AtomicBot-ai#8 0x0001009b8afc in common_chat_templates_apply_jinja(common_chat_templates const*, common_chat_templates_inputs const&) chat.cpp:1992
    AtomicBot-ai#9 0x0001009b533c in common_chat_templates_apply(common_chat_templates const*, common_chat_templates_inputs const&) chat.cpp:2074
    AtomicBot-ai#10 0x000100810120 in llamacpp_apply_chat_template+0x724 (predict_oai-98384e17fb94e863:arm64+0x100090120)
    ...

==45482==Register values:
 x[0] = 0x00006020004147f8   x[1] = 0x00006080000013c8   x[2] = 0x0000000000000000   x[3] = 0x0000604006289738
 x[4] = 0x0000000000000002   x[5] = 0x0000000000000001   x[6] = 0x04034000004b4000   x[7] = 0x0000000000000001
 x[8] = 0xbebebebebebebebe   x[9] = 0x17d7d7d7d7d7d7d7  x[10] = 0x00000c04000828ff  x[11] = 0x0000000000000001
x[12] = 0x000000002018d383  x[13] = 0x0000000000000000  x[14] = 0xfa0000000000fafa  x[15] = 0x000010700001ffff
x[16] = 0x000000019dc012c0  x[17] = 0x00000001021284f8  x[18] = 0x0000000000000000  x[19] = 0x00000001700acdc0
x[20] = 0x0000000000000002  x[21] = 0x000000002018d384  x[22] = 0x16dd16fd2e731151  x[23] = 0x0000007000020000
x[24] = 0x0000000100c69c08  x[25] = 0x0000000100c69c20  x[26] = 0x00006080000013c7  x[27] = 0x0000000100c69c00
x[28] = 0x00000001700acd60     fp = 0x00000001700aceb0     lr = 0x0000000100abce30     sp = 0x00000001700acd60
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV __hash_table:1565 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void*>*>, bool> std::__1::__hash_table<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
Thread T5 created by T0 here:
    #0 0x0001020b99d4 in pthread_create+0x5c (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x359d4)
    AtomicBot-ai#1 0x000100873910 in std::sys::pal::unix::thread::Thread::new::h77254fdd87a28e05+0x118 (predict_oai-98384e17fb94e863:arm64+0x1000f3910)
    AtomicBot-ai#2 0x0001007c7a1c in test::run_test::haeb3c2bcd5ed6cf6+0x76c (predict_oai-98384e17fb94e863:arm64+0x100047a1c)
    AtomicBot-ai#3 0x0001007aedb0 in test::console::run_tests_console::he9d142d704f3a986+0x149c (predict_oai-98384e17fb94e863:arm64+0x10002edb0)
    AtomicBot-ai#4 0x0001007c5758 in test::test_main::hf86a5e20735245b9+0x118 (predict_oai-98384e17fb94e863:arm64+0x100045758)
    AtomicBot-ai#5 0x0001007c5da0 in test::test_main_static::h61ee9c8fd30abca0+0x54 (predict_oai-98384e17fb94e863:arm64+0x100045da0)
    ...

==45482==ABORTING
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@TheTom @claude
fix: TURBO_D=128 independent of QK_TURBO3, file turbo4 bugs AtomicBot…
40c4bdc 
…-ai#29

Codex post-commit review found:
1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes
2. SET_ROWS kernel turbo3-specific but instantiated for turbo4
3. Tail block drop for non-128 head dims

Fixed AtomicBot-ai#3 (TURBO_D). AtomicBot-ai#1 and AtomicBot-ai#2 don't affect turbo3+dk128 path.

Co-Authored-By: tturney@psyguard.ai
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@TheTom @claude
final: 12 approaches tested, 4-mag LUT is the hardware limit
3d8b22e 
Complete experiment log:
  AtomicBot-ai#1  4-mag LUT:           15.1 at 8K (BEST, +38%)
  AtomicBot-ai#2  Batched extract:     13.7 (+25%)
  AtomicBot-ai#3  Inline FA block:     13.5 (I-cache pressure)
  AtomicBot-ai#4  Deferred norm:       12.9 (loses ILP)
  AtomicBot-ai#5  2-pair half2:        12.0 (ternary overhead)
  AtomicBot-ai#6  Select chain:        11.9 (branches kill)
  AtomicBot-ai#7  Bit-arithmetic:      11.6 (ALU too heavy)
  AtomicBot-ai#8  FMA branchless:      11.4 (ALU still too heavy)
  AtomicBot-ai#9  Named-reg ternary:   10.3 (branches worst)
  AtomicBot-ai#10 Main (8-LUT):        10.95 (baseline)
  AtomicBot-ai#11 Non-vec FA:          10.2 (wrong kernel)
  Ceiling:                 24.5 (no dequant)

Apple8 hardware truth:
  1 divergent constant read < 7 ALU ops (even with fma)
  Branches cost MORE than divergent constant reads
  Array indexing ALWAYS spills on Metal
  4 constant addresses is the sweet spot

The 4-mag LUT is the dequant-level ceiling on Apple Silicon.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: tturney@psyguard.ai
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@Ooooze
Merge pull request AtomicBot-ai#3 from AtomicBot-ai/feature/gemma-mtp
0c94258 
Feature/gemma mtp
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 12, 2026
fukuro
feat: Pascal CUDA MMVQ Optimization (PR ggml-org#25479) — manuell por…
5c18849 
…tiert

AtomicBot-ai#3 Pascal CUDA MMVQ Optimization:
- MMVQ_PARAMETERS_PASCAL_DP4A zum Enum hinzugefügt
- get_device_table_id(): Pascal CC 6.1/6.2 Detection (DP4A bis Volta)
- calc_nwarps(): 2 Warps statt 4 für single-token decode auf Pascal
  (bandwidth-bound auf kleinen SMs)
- calc_rows_per_block(): Pascal in GENERIC/GCN/TURING-Gruppe aufgenommen
- +3-6% decode auf GTX 1060/1070 (Pascal Consumer)
- Minimaler Change (+14 -3 Zeilen), Pascal-spezifisch, keine Auswirkung
  auf neuere GPUs (Volta+) oder AMD

PR ggml-org#25479 ist Draft/open — manuelle Portierung wegen TurboQuant-
MMVQ-Modifikationen (eigene Quantisierungstypen, erweiterte Tabellen)
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 12, 2026
fukuro
docs: AtomicBot-ai#3 Pascal MMVQ +8.9% Generation benchmark, AtomicBo…
bb1ca43 
…t-ai#13 Two-Tier rejected

AtomicBot-ai#3 Pascal CUDA MMVQ Benchmark (Styx, GTX 1070, E2B MoE):
- Baseline: 65.1 t/s Generation, 908.9 t/s Prompt
- Mit MMVQ: 70.9 t/s Generation (+8.9%), 915.3 t/s Prompt (+0.7%)
- Besser als PR-Versprechen (+3-6%)

AtomicBot-ai#13 Two-Tier Expert Cache: nicht implementieren
- Alle 4 PRs closed ohne Merge
- thecodacus Memory Pinning deckt Tier 2 bereits ab
- Geringer ROI für Pascal (compute-bound)
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 12, 2026
fukuro
docs: AtomicBot-ai#7 Vulkan FA Refactor bereits integriert, ROADMAP-A…
26b7469 
…udit abgeschlossen

AtomicBot-ai#7 Vulkan FA Refactor (PR ggml-org#19625): bereits im Fork (Commit 66e999e)
- get_fa_tuning_params_scalar(), row_split, shmem_staging
- Q caching in registers, vendor-specific Br selection
- 28 Referenzen im Code

ROADMAP-Audit Ergebnis:
- M1: ✅ abgeschlossen
- M2: ✅ evaluiert (AtomicBot-ai#6✅, AtomicBot-ai#7✅ bereits integriert, AtomicBot-ai#9❌, AtomicBot-ai#10❌)
- M3: ⏳ teilweise (AtomicBot-ai#3✅, AtomicBot-ai#13❌, AtomicBot-ai#14⏭️, AtomicBot-ai#15 offen)
- M4-M6: Noch nicht begonnen (Tier 2-4, 1+ Wochen Aufwand)
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 12, 2026
fukuro
docs: ROADMAP-Audit Phase 3 — 5 weitere Items bereits integriert
fa07c53 
Systematische Prüfung aller ROADMAP-Items gegen Fork-Code:
- AtomicBot-ai#11 EAGLE-3 (PR ggml-org#18039): bereits integriert (Commit 57774253c)
- AtomicBot-ai#12 Coopmat2 (PR ggml-org#19075): bereits integriert (flash_attn_cm2.comp, SPV generiert)
- AtomicBot-ai#20 Tensor Parallelism (PR ggml-org#19378): bereits integriert (Commit d850df3)
- AtomicBot-ai#28 Adaptive MTP (PR ggml-org#22931): Fork hat eigene Implementierung
  (LLAMA_MTP_SKIP_STREAK_THRESHOLD), PR closed/inkompatibel

Meilenstein-Status:
- M1: ✅ abgeschlossen
- M2: ✅ evaluiert (AtomicBot-ai#6✅, AtomicBot-ai#7✅, AtomicBot-ai#12✅ bereits integriert, AtomicBot-ai#9❌, AtomicBot-ai#10❌)
- M3: ⏳ teilweise (AtomicBot-ai#3✅, AtomicBot-ai#13❌, AtomicBot-ai#14⏭️, AtomicBot-ai#15 offen)
- M4: ✅ abgeschlossen (AtomicBot-ai#11✅, AtomicBot-ai#28✅ eigene Implementierung)
- M5: ⏳ teilweise (AtomicBot-ai#12✅, AtomicBot-ai#20✅ bereits integriert, AtomicBot-ai#21 offen)
- M6: ☐ offen (Tier 4 Forschung)
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 12, 2026
fukuro
docs: Zeit-Schätzungen kalibriert — 2-3x Multiplikator auf Subagent-S…
fa6c83f 
…chätzungen

Kalibrierung aus 14h Solo-Session (2026-07-11/12):
- AtomicBot-ai#3 Pascal MMVQ: geschätzt '2-3 Tage', tatsächlich ~3-4h mit PR-Vorlage
- AtomicBot-ai#32 L1 Cache: geschätzt '1-2h', tatsächlich ~1h (trivialer Flag)
- Bulk der Zeit ging in Evaluation, Doku, Build/Benchmark-Zyklen

Realität vs. Subagent-Schätzungen:
- Remote-Build: 5-15 min/Zyklus
- Benchmark: 5-10 min/Run + Server-Management
- Doku + Commit: 15-30 min/Item
- Subagent-Latenz: 2-5 min/Round-Trip

Neue Tier-Definitionen:
- Tier 1: < 2 Tage Solo-Agent (war < 1 Woche)
- Tier 2: 2-6 Wochen (war 1-3 Wochen)
- Tier 3: 6-12 Wochen (war 3-6 Wochen)
- Tier 4: 3+ Monate (war 6+ Wochen)

Alle 55 Items mit kalibrierten Schätzungen aktualisiert.
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 19, 2026
fukuro
docs: Research-Sweep AtomicBot-ai#3 — 21 neue Optimierungs-Ansätze (2…
ca969ed 
…026-07-13), ROADMAP AtomicBot-ai#56-TheTom#76, M3 abgeschlossen
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 19, 2026
fukuro
docs: SNAPSHOT aktualisiert mit UMA Cached + Research-Sweep AtomicBot…
c24991d 
…-ai#3
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 19, 2026
fukuro
fix: AtomicBot-ai#61 review — invalidate() clears all per-blk state +…
01fd10a 
… maps (UAF, state leaks)

P0 AtomicBot-ai#1 (Use-After-Free): invalidate() cleared only role_base but left
blk_down_base/blk_down_kb/blk_pair_pool/blk_down_pool stale. After a host
buffer free, moe_cache_backfill_next() would build jobs reading from freed
memory. Now clears ALL per-blk pointers that point into the freed range and
resets pool indices to -1 for affected blocks.

P1 AtomicBot-ai#3 (State leaks on model unload): glu_learn, learn_gate_dst,
learn_up_dst, redirect, g_disc (seen/pending/stable_count/any_repeat) and
backfill cursor survived invalidate() — a new model loaded at the same
addresses would hit stale learned entries and pool decisions. All cleared
now.

P2 AtomicBot-ai#7: removed unused 'self' variable in moe_cache_begin(). Added comment
documenting the owner-lock limitation (P1 AtomicBot-ai#4: not reset on model reload —
acceptable for one-model-per-process, the common case).
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 22, 2026
fukuro
fix: /cancel + dev_reset P1/P2 Review-Fixes
e3d9103 
P1 AtomicBot-ai#1: /cancel false-positive bei nicht-existierender task_id
  - METRICS-Check jetzt für ALLE task_ids (nicht nur bei leerem Body)
  - task_id wird gegen laufende Slots validiert vor Cancel-Post
  - Nicht-existierende task_id → {cancelled: false, error: 'task not found'}
  - Leerer Body ohne laufende Tasks → {cancelled: false, message: 'no running tasks'}
  - Zusätzlich: ältester Task (nach start_time) statt niedrigster Slot-Index

P1 AtomicBot-ai#2: ggml_backend_cuda_device_reset thread-safety
  - device_mutex Lock hinzugefügt (wie ggml_backend_cuda_device_get_memory)
  - active_count > 0 → Reset verweigert (verhindert Context-Crash)
  - cudaGetLastError-Details in GGML_LOG_WARN

P1 AtomicBot-ai#3: Test-Skript — Cancel-Wirkung verifiziert
  - Test 3: Stream muss abgebrochen sein (aborted=True oder wenige chunks)
  - Test 3b neu: nicht-existierende task_id → cancelled=false + error
  - Test akzeptiert nicht mehr normal beendeten Stream als Erfolg

P1 AtomicBot-ai#4: dev_reset Rückgabewert nicht ignorieren
  - SRV_WRN bei fehlgeschlagenem Reset mit Device-Name

P2 AtomicBot-ai#7: proxy_post try/catch bei leerem/ungültigem Body
  - Statt 500-Exception → 400 'Invalid JSON body'
  - res_err() statt nicht-existenter .error() Methode

P2 AtomicBot-ai#8: Ältester Task statt niedrigster Slot-Index (in P1 AtomicBot-ai#1 fix enthalten)
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 22, 2026
fukuro
fix: Review Round 2 — start_time in slot.to_json(), cudaSetDevice log…
3379449 
…ging, cstdint

P1 AtomicBot-ai#1: start_time Feld in server_slot::to_json() hinzugefügt
  - t_start_process_prompt (int64_t, Mikrosekunden) als 'start_time' exportiert
  - post_cancel liest jetzt int64_t statt int — kein Überlauf mehr
  - Ältester-Task-Auswahl funktioniert jetzt tatsächlich (vorher immer 0 → erster Slot)

P1 AtomicBot-ai#2: dev_reset failure — Kommentar erklärt warum Sleep trotzdem betreten wird
  - Modell ist bereits zerstört wenn Reset fehlschlägt → Abbruch = unrecoverable
  - load_model() beim Aufwecken wird OOM-failen wenn VRAM noch belegt → geloggt

P1 AtomicBot-ai#3: cudaSetDevice Fehlerlogging mit cudaGetLastError() korrigiert
  - Return-Wert von cudaSetDevice prüfen, cudaGetLastError() für Fehlermeldung
  - Vorher: cudaGetLastError() konnte veralteten/no-error Zustand loggen

P2 AtomicBot-ai#4: <cstdint> explizit inkludiert für INT64_MAX (portabilität)
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Aug 2, 2026
fukuro
fix(moe-cache): review-swe P1 fixes — queued-slot detection, off-by-o…
d9346c0 
…ne, invalidate

P1 AtomicBot-ai#1: Set-assoc lookup erkennt jetzt queued-Slots (key match bei
queued=true) → kein Duplikat-Insert mehr im selben Set. Backfill
entsprechend angepasst.
P1 AtomicBot-ai#2: Workload window reset off-by-one — reset nach wsize Aufrufen,
nicht am Start des wsize-ten Aufrufs.
P1 AtomicBot-ai#4: invalidate() cleart jetzt set_lru_head/tail/n_used und
window_count in allen Pools (wie trim() es schon tut).
P1 AtomicBot-ai#5: Kommentar 'Aufrunden' → 'Abrunden' (korrigiert).

Review: review-swe Subagent, 5 P1 + 7 P2 Issues gefunden.
P1 AtomicBot-ai#3 (pool-init locking) ist bestehendes Problem, nicht neu.
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Aug 2, 2026
fukuro
fix(moe-cache): P1#3 pool-init locking + P2 workload tie-breaker + P3…
f942ef2 
… ROADMAP

P1 AtomicBot-ai#3: Pool-Alloc in moe_cache_begin unter g.mu Lock — verhindert Data Race
mit Worker/Backfill-Threads die d.pools[pi] lesen. cudaMalloc unter Lock
akzeptabel (nur bei Pool-Erstellung, nicht im Hot Path).

P2: Workload-Eviction Tie-Breaker — bei gleichem workload_score den LRU-
ältesten Slot evicten (cur_tick - last_access). Verhindert deterministische
Slot-0-Eviction nach Window-Reset.

P3: ROADMAP 'MoE-Cache erschöpft' Sektion — fasst AtomicBot-ai#69/AtomicBot-ai#71/AtomicBot-ai#18 Ergebnisse
zusammen. LRU ist optimal bei 128-Expert 24:1 Oversubscription.
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Aug 2, 2026
fukuro
research: Monatliche Optimierungs-Recherche 2026-07-26 (Research-Sweep …
311790c 
…AtomicBot-ai#5)

4 parallele Subagents (Vulkan/AMD, CUDA/MoE, arXiv Papers, Multi-GPU/Batching).
51 Items gesammelt, dedupliziert, 5 Quick-Wins verifiziert.

Verifikation: 3/5 Quick-Wins bereits im Fork (ggml-org#15524=AtomicBot-ai#6, ggml-org#16391=--cram, ggml-org#19754= warmup).
PR ggml-org#25479 (Pascal MMVQ) bereits als AtomicBot-ai#3 ✅. PR ggml-org#22887 (4K per Iter) bereits im Fork.

9 neue ROADMAP-Items hinzugefügt:
- TheTom#89 Transfer Queue AMD RDNA3 (PR ggml-org#19976)
- TheTom#90 MUL_MAT_ID Non-Square Tile (Discussion ggml-org#22598)
- TheTom#91 Internal AllReduce Kernel (PR ggml-org#22299)
- TheTom#92 RateQuant KV Cache (arXiv:2605.06675)
- TheTom#93 InnerQ KV Cache (arXiv:2602.23200)
- TheTom#94 FineMoE Expert Offloading (arXiv:2502.05370)
- TheTom#95 Shared Expert Aux Stream (TensorRT-LLM)
- TheTom#96 Speculative Checkpointing (PR ggml-org#19493)
- TheTom#97 Backend-agnostic TP Meta Device (PR ggml-org#19378)

Vollständiger Report: docs/fork/2026-07-26_OPTIMIZATION_RESEARCH.md
NighmareGit pushed a commit to NighmareGit/atomic-llama-cpp-turboquant that referenced this pull request Aug 8, 2026
@timkhronos @danielhanchen
@CISC @ngxson
model: Add MiniMax-M3 (MSA: MiniMax Sparse Attention) support (ggml-o…
b1d4c65 
…rg#24908)

* Add preliminary MiniMax-M3 support

Text-only port that re-uses existing components: MiniMax-M2 style GQA with
per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and
routed/shared experts, and swigluoai activation. Sparse attention is not
yet supported (dense fallback); vision tower and MTP heads are dropped.

* MiniMax-M3 vision tower (mmproj + clip graph)

* Delete m3_vision_ref.py

* Update clip.cpp

* MSA

* Update constants.py

* Update minimax.py

* Cache creation. Working withotu flash attention

* Added flash attention for sparse layers

* Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx

* Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking

* Implement sparse attention calc out of stock ops.

* Fix a cache allocation and cont issue

* Fixed -fa auto crash, flagged debug spots

* Delete vocab.json

* Delete model.safetensors.index.json

* Delete generation_config.json

* Delete Minimax directory

* Handled multi stream case to fall back on Dense Attention

* Development scaffolding cleanup. No functional change to the decode or
4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the
selection-parity validation.

* Remove redundant comment from minimax-m3.cpp

* Changed 3 Gelu Ops for vision into Gelu_erf ops

* Assert that n_kv is multiple of 128

* Rename MSA index tensors to indexer convention

Note: All GGUFs generated before this change will need to be regenerated.

* Fix incorrect Assert

* Review driven changes (AtomicBot-ai#3)

* Remove comment from conversion minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove whitespaces from constants.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Tighten comment in minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* inherit MiniMax-M3 from MiniMax-M2

* drop dead text_config fallbacks

* Add indexer writer methods

* Reuse LLM_FFN_SWIGLU_OAI_MOE

* Remove duplicate  indexer setters, add only block_size/local_blocks, follow value naming convention

* Fix conversion error /gguf_writer.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update gguf-py/gguf/gguf_writer.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update conversion/minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update conversion/minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove whitespace in src/llama-kv-cache.cpp

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove Whitespace in Update src/llama-model.h

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove whitespace in src/llama-hparams.h

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* remove multimodal code upon maintainer request. Will be made as a separate PR

* Whitespace clean in tensor_mapping.py

* Log cache size on launch, block ctx shift, support prompt caching

Log indexer cache size on launch

Disallow ctx shift

Support prompt caching

* Update minimax-m3.cpp

* Optimize implementation, add multi stream support. 

Fully rewrote minimax-m3.cpp for speed and buffer size gains:

Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3]

Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill

Decode: ~25 nodes/layer vs ~50, no per-group concats/conts

Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection

can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token)

In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k

Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq

Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support.

* set default cache type to F32

* Fix potential DSA double indexer cache  allocation bug, only allocate in-cache k_idx for archs that opt in

* remove F16 downcasts in MSA attention, force F32 indexer score accum

* Add Minimax eos to llama vocab

* Guard edge case where idx cache can become stale after a tail trim

* Update llama-kv-cache.h

* Update llama-kv-cache.cpp

* Update llama-kv-cache.cpp

* Update llama-kv-cache.h

* Update llama-kv-cache.cpp

* Review driven changes

* style fix

* indexer hparams are required

* fix tests

* fix lint

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
NighmareGit pushed a commit to NighmareGit/atomic-llama-cpp-turboquant that referenced this pull request Aug 8, 2026
@timkhronos @danielhanchen
@CISC @ngxson
mtmd: Add Vision Support for Minimax-M3 (ggml-org#25113)
3d1c3a8 
* Add preliminary MiniMax-M3 support

Text-only port that re-uses existing components: MiniMax-M2 style GQA with
per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and
routed/shared experts, and swigluoai activation. Sparse attention is not
yet supported (dense fallback); vision tower and MTP heads are dropped.

* MiniMax-M3 vision tower (mmproj + clip graph)

* Delete m3_vision_ref.py

* Update clip.cpp

* MSA

* Update constants.py

* Update minimax.py

* Cache creation. Working withotu flash attention

* Added flash attention for sparse layers

* Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx

* Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking

* Implement sparse attention calc out of stock ops.

* Fix a cache allocation and cont issue

* Fixed -fa auto crash, flagged debug spots

* Delete vocab.json

* Delete model.safetensors.index.json

* Delete generation_config.json

* Delete Minimax directory

* Handled multi stream case to fall back on Dense Attention

* Development scaffolding cleanup. No functional change to the decode or
4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the
selection-parity validation.

* Remove redundant comment from minimax-m3.cpp

* Changed 3 Gelu Ops for vision into Gelu_erf ops

* Assert that n_kv is multiple of 128

* Rename MSA index tensors to indexer convention

Note: All GGUFs generated before this change will need to be regenerated.

* Fix incorrect Assert

* Review driven changes (AtomicBot-ai#3)

* Remove comment from conversion minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove whitespaces from constants.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Tighten comment in minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* inherit MiniMax-M3 from MiniMax-M2

* drop dead text_config fallbacks

* Add indexer writer methods

* Reuse LLM_FFN_SWIGLU_OAI_MOE

* Remove duplicate  indexer setters, add only block_size/local_blocks, follow value naming convention

* Fix conversion error /gguf_writer.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update gguf-py/gguf/gguf_writer.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update gguf-py/gguf/tensor_mapping.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update conversion/minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update conversion/minimax.py

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove whitespace in src/llama-kv-cache.cpp

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove Whitespace in Update src/llama-model.h

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Remove whitespace in src/llama-hparams.h

Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>

* Update minimax_m3.cpp

Rewrite code comment based on feedback and to better reflect the actual architecture, and reuse existing build_vit

* Rename minimax_m3.cpp to minimax-m3.cpp

* Update CMakeLists.txt

* Remove debug code from clip.cpp

* Update clip.cpp

* Update comments in tools/mtmd/models/minimax-m3.cpp

* Permute Q/K at conversion, drop precomputed sin/cos

* Log cache size on launch, block ctx shift, support prompt caching

Log indexer cache size on launch

Disallow ctx shift

Support prompt caching

* Update minimax-m3.cpp

* Optimize implementation, add multi stream support. 

Fully rewrote minimax-m3.cpp for speed and buffer size gains:

Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3]

Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill

Decode: ~25 nodes/layer vs ~50, no per-group concats/conts

Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection

can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token)

In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k

Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq

Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support.

* set default cache type to F32

* Fix potential DSA double indexer cache  allocation bug, only allocate in-cache k_idx for archs that opt in

* remove F16 downcasts in MSA attention, force F32 indexer score accum

* Add Minimax eos to llama vocab

* Guard edge case where idx cache can become stale after a tail trim

* Update llama-kv-cache.h

* Update llama-kv-cache.cpp

* Update llama-kv-cache.cpp

* Update llama-kv-cache.h

* Change resize Pad to none, resize alg to Bicubic Pillow

* Review driven changes

* Update llama-kv-cache.cpp

* rm unrotated pos_t

* fused rope w + pad

* rename merge --> merger for consistency

* add review skill for mtmd

* graph should use hparams n_merge

* fix lint

---------

Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation examples model python script server testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Ooooze