'.\!start - ling.py'
Starting llama-server...
0.00.244.436 I cmn common_param: common_params_print_info: build 10696 (cd5609390) with Clang 22.1.8 for Windows AMD64
0.00.244.439 I cmn common_param: common_params_print_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
0.00.244.441 I cmn common_param: device_info:
0.00.311.086 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 5070 (12226 MiB, 11035 MiB free)
0.00.311.094 I cmn common_param: - CPU : AMD Ryzen 7 5700X3D 8-Core Processor (57244 MiB, 51295 MiB free)
0.00.325.146 I cmn common_param: system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 1200 | FORCE_MMQ = 1 | USE_GRAPHS = 1 | FA_ALL_QUANTS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.343.834 I srv init: using 15 threads for HTTP server
0.00.347.088 W srv llama_server: -----------------
0.00.347.091 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.347.092 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.347.092 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.347.093 W srv llama_server: -----------------
0.00.348.990 I srv start: binding port with default address family
0.00.351.847 I srv load_model: loading model 'D:\Ling-3.0-flash-AD-IQ3_S-00001-of-00002.gguf'
0.00.351.850 I srv load_model: local path 'D:\Ling-3.0-flash-AD-IQ3_S-00001-of-00002.gguf'
0.00.352.132 I cmn common_init_: fitting params to device memory ...
0.00.352.135 I cmn common_init_: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.352.645 I common_params_fit_impl: getting device memory data for initial parameters:
0.00.776.765 I common_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
0.00.776.768 I common_memory_breakdown_print: | - CUDA0 (RTX 5070) | 12226 = 10959 + ( 9581 = 6831 + 998 + 1752) + -8314 |
0.00.776.769 I common_memory_breakdown_print: | - Host | 49057 = 48310 + 0 + 747 |
0.00.820.808 I common_params_fit_impl: projected to use 9581 MiB of device memory vs. 10959 MiB of free device memory
0.00.820.811 I common_params_fit_impl: will leave 1377 >= 1024 MiB of free device memory, no changes needed
0.00.820.937 I common_fit_params: successfully fit params to free device memory
0.00.820.939 I common_fit_params: fitting params to free memory took 0.47 seconds
0.00.867.519 I llama_model_loader: additional 1 GGUFs metadata loaded.
0.00.867.523 I llama_model_loader: loaded meta data with 58 key-value pairs and 917 tensors from D:\Ling-3.0-flash-AD-IQ3_S-00001-of-00002.gguf (version GGUF V3 (latest))
0.00.867.547 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.00.867.549 I llama_model_loader: - kv 0: general.architecture str = bailingmoe3
0.00.867.550 I llama_model_loader: - kv 1: general.type str = model
0.00.867.551 I llama_model_loader: - kv 2: general.name str = Ling 3.0 Flash
0.00.867.551 I llama_model_loader: - kv 3: general.size_label str = 512x3.9B
0.00.867.552 I llama_model_loader: - kv 4: general.license str = mit
0.00.867.560 I llama_model_loader: - kv 5: general.tags arr[str,1] = ["text-generation"]
0.00.867.561 I llama_model_loader: - kv 6: bailingmoe3.block_count u32 = 42
0.00.867.562 I llama_model_loader: - kv 7: bailingmoe3.context_length u32 = 131072
0.00.867.563 I llama_model_loader: - kv 8: bailingmoe3.embedding_length u32 = 2560
0.00.867.563 I llama_model_loader: - kv 9: bailingmoe3.feed_forward_length u32 = 6144
0.00.867.564 I llama_model_loader: - kv 10: bailingmoe3.attention.head_count u32 = 32
0.00.867.570 I llama_model_loader: - kv 11: bailingmoe3.attention.head_count_kv arr[i32,42] = [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, ...
0.00.867.573 I llama_model_loader: - kv 12: bailingmoe3.rope.freq_base f32 = 6000000.000000
0.00.867.575 I llama_model_loader: - kv 13: bailingmoe3.attention.layer_norm_rms_epsilon f32 = 0.000001
0.00.867.575 I llama_model_loader: - kv 14: bailingmoe3.expert_count u32 = 512
0.00.867.576 I llama_model_loader: - kv 15: bailingmoe3.expert_used_count u32 = 8
0.00.867.576 I llama_model_loader: - kv 16: bailingmoe3.expert_group_count u32 = 8
0.00.867.576 I llama_model_loader: - kv 17: bailingmoe3.expert_group_used_count u32 = 4
0.00.867.577 I llama_model_loader: - kv 18: bailingmoe3.expert_gating_func u32 = 2
0.00.867.577 I llama_model_loader: - kv 19: bailingmoe3.attention.key_length u32 = 576
0.00.867.578 I llama_model_loader: - kv 20: bailingmoe3.attention.value_length u32 = 512
0.00.867.578 I llama_model_loader: - kv 21: bailingmoe3.vocab_size u32 = 157184
0.00.867.579 I llama_model_loader: - kv 22: bailingmoe3.ssm.conv_kernel u32 = 4
0.00.867.579 I llama_model_loader: - kv 23: bailingmoe3.kda.head_dim u32 = 128
0.00.867.580 I llama_model_loader: - kv 24: bailingmoe3.attention.kv_lora_rank u32 = 512
0.00.867.580 I llama_model_loader: - kv 25: bailingmoe3.attention.key_length_mla u32 = 192
0.00.867.581 I llama_model_loader: - kv 26: bailingmoe3.attention.value_length_mla u32 = 128
0.00.867.581 I llama_model_loader: - kv 27: bailingmoe3.rope.dimension_count u32 = 64
0.00.867.582 I llama_model_loader: - kv 28: bailingmoe3.leading_dense_block_count u32 = 2
0.00.867.582 I llama_model_loader: - kv 29: bailingmoe3.expert_feed_forward_length u32 = 768
0.00.867.583 I llama_model_loader: - kv 30: bailingmoe3.expert_shared_feed_forward_length u32 = 768
0.00.867.583 I llama_model_loader: - kv 31: bailingmoe3.expert_shared_count u32 = 1
0.00.867.584 I llama_model_loader: - kv 32: bailingmoe3.expert_weights_scale f32 = 2.500000
0.00.867.585 I llama_model_loader: - kv 33: bailingmoe3.expert_weights_norm bool = true
0.00.867.585 I llama_model_loader: - kv 34: bailingmoe3.nextn_predict_layers u32 = 1
0.00.867.586 I llama_model_loader: - kv 35: tokenizer.ggml.model str = gpt2
0.00.867.586 I llama_model_loader: - kv 36: tokenizer.ggml.pre str = bailingmoe2
0.00.895.333 I llama_model_loader: - kv 37: tokenizer.ggml.tokens arr[str,157184] = ["!", "\"", "#", "$", "%", "&", "'", ...
0.00.904.601 I llama_model_loader: - kv 38: tokenizer.ggml.token_type arr[i32,157184] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
0.00.932.119 I llama_model_loader: - kv 39: tokenizer.ggml.merges arr[str,156635] = ["Ġ Ġ", "Ġ t", "i n", "Ġ a", "h e...
0.00.932.124 I llama_model_loader: - kv 40: tokenizer.ggml.bos_token_id u32 = 156891
0.00.932.126 I llama_model_loader: - kv 41: tokenizer.ggml.eos_token_id u32 = 156895
0.00.932.127 I llama_model_loader: - kv 42: tokenizer.ggml.padding_token_id u32 = 156892
0.00.932.128 I llama_model_loader: - kv 43: tokenizer.ggml.add_bos_token bool = false
0.00.932.128 I llama_model_loader: - kv 44: tokenizer.ggml.add_eos_token bool = false
0.00.932.132 I llama_model_loader: - kv 45: tokenizer.chat_template str = {#- Bailing V3 chat template -#}\n{#- ...
0.00.932.135 I llama_model_loader: - kv 46: bailingmoe3.kda.gate_lower_bound f32 = -5.000000
0.00.932.135 I llama_model_loader: - kv 47: general.quantization_version u32 = 2
0.00.932.136 I llama_model_loader: - kv 48: general.file_type u32 = 26
0.00.932.137 I llama_model_loader: - kv 49: quantize.imatrix.file str = /root/work/ling3-imatrix-bf16.gguf
0.00.932.137 I llama_model_loader: - kv 50: quantize.imatrix.dataset str = /root/work/calib/calib_train.txt
0.00.932.138 I llama_model_loader: - kv 51: quantize.imatrix.entries_count u32 = 573
0.00.932.138 I llama_model_loader: - kv 52: quantize.imatrix.chunks_count u32 = 522
0.00.932.139 I llama_model_loader: - kv 53: split.no u16 = 0
0.00.932.140 I llama_model_loader: - kv 54: split.tensors.count i32 = 917
0.00.932.140 I llama_model_loader: - kv 55: split.count u16 = 2
0.00.932.158 I llama_model_loader: - kv 56: bailingmoe3.swiglu_clamp_exp arr[f32,42] = [0.000000, 0.000000, 0.000000, 0.0000...
0.00.932.173 I llama_model_loader: - kv 57: bailingmoe3.swiglu_clamp_shexp arr[f32,42] = [0.000000, 0.000000, 0.000000, 0.0000...
0.00.932.174 I llama_model_loader: - type f32: 382 tensors
0.00.932.174 I llama_model_loader: - type f16: 1 tensors
0.00.932.175 I llama_model_loader: - type q8_0: 414 tensors
0.00.932.175 I llama_model_loader: - type iq3_s: 105 tensors
0.00.932.176 I llama_model_loader: - type iq4_xs: 15 tensors
0.00.932.176 I print_info: file format = GGUF V3 (latest)
0.00.932.177 I print_info: file type = IQ3_S - 3.4375 bpw
0.00.932.178 I print_info: file size = 53.85 GiB (3.72 BPW)
0.00.932.210 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 5070) (0000:06:00.0) - 11035 MiB free
0.00.997.223 I load: 0 unused tokens
0.01.004.737 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.01.004.739 I load: printing all EOG tokens:
0.01.004.739 I load: - 156892 ('<|endoftext|>')
0.01.004.740 I load: - 156895 ('<|role_end|>')
0.01.004.882 I load: special tokens cache size = 262
0.01.031.211 I load: token to piece cache size = 1.0009 MB
0.01.031.220 I print_info: arch = bailingmoe3
0.01.031.221 I print_info: vocab_only = 0
0.01.031.221 I print_info: no_alloc = 0
0.01.031.222 I print_info: n_ctx_train = 131072
0.01.031.222 I print_info: n_embd_inp = 2560
0.01.031.223 I print_info: n_embd = 2560
0.01.031.223 I print_info: n_embd_out = 2560
0.01.031.224 I print_info: n_layer = 42
0.01.031.224 I print_info: n_layer_all = 42
0.01.031.232 I print_info: n_head = 32
0.01.031.243 I print_info: n_head_kv = [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1]
0.01.031.243 I print_info: n_rot = 64
0.01.031.244 I print_info: n_swa = 0
0.01.031.244 I print_info: is_swa_any = 0
0.01.031.245 I print_info: n_embd_head_k = 576
0.01.031.245 I print_info: n_embd_head_v = 512
0.01.031.255 I print_info: n_gqa = [0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0, 32, 0, 0, 0, 0, 0, 32]
0.01.031.265 I print_info: n_embd_k_gqa = [0, 0, 0, 0, 0, 576, 0, 0, 0, 0, 0, 576, 0, 0, 0, 0, 0, 576, 0, 0, 0, 0, 0, 576, 0, 0, 0, 0, 0, 576, 0, 0, 0, 0, 0, 576, 0, 0, 0, 0, 0, 576]
0.01.031.274 I print_info: n_embd_v_gqa = [0, 0, 0, 0, 0, 512, 0, 0, 0, 0, 0, 512, 0, 0, 0, 0, 0, 512, 0, 0, 0, 0, 0, 512, 0, 0, 0, 0, 0, 512, 0, 0, 0, 0, 0, 512, 0, 0, 0, 0, 0, 512]
0.01.031.276 I print_info: f_norm_eps = 0.0e+00
0.01.031.277 I print_info: f_norm_rms_eps = 1.0e-06
0.01.031.277 I print_info: f_clamp_kqv = 0.0e+00
0.01.031.278 I print_info: f_max_alibi_bias = 0.0e+00
0.01.031.278 I print_info: f_logit_scale = 0.0e+00
0.01.031.279 I print_info: f_attn_scale = 0.0e+00
0.01.031.279 I print_info: f_attn_value_scale = 0.0000
0.01.031.281 I print_info: n_ff = 6144
0.01.031.282 I print_info: n_expert = 512
0.01.031.282 I print_info: n_expert_used = 8
0.01.031.282 I print_info: n_expert_groups = 8
0.01.031.283 I print_info: n_group_used = 4
0.01.031.283 I print_info: causal attn = 1
0.01.031.284 I print_info: pooling type = -1
0.01.031.284 I print_info: rope type = 0
0.01.031.285 I print_info: rope scaling = linear
0.01.031.286 I print_info: freq_base_train = 6000000.0
0.01.031.287 I print_info: freq_scale_train = 1
0.01.031.287 I print_info: n_ctx_orig_yarn = 131072
0.01.031.288 I print_info: rope_yarn_log_mul = 0.0000
0.01.031.288 I print_info: rope_finetuned = unknown
0.01.031.289 I print_info: model type = 124B.A5B
0.01.031.290 I print_info: model params = 124.41 B
0.01.031.299 I print_info: general.name = Ling 3.0 Flash
0.01.031.300 I print_info: vocab type = BPE
0.01.031.300 I print_info: n_vocab = 157184
0.01.031.300 I print_info: n_merges = 156635
0.01.031.301 I print_info: BOS token = 156891 '<|startoftext|>'
0.01.031.301 I print_info: EOS token = 156895 '<|role_end|>'
0.01.031.302 I print_info: EOT token = 156892 '<|endoftext|>'
0.01.031.302 I print_info: PAD token = 156892 '<|endoftext|>'
0.01.031.303 I print_info: LF token = 198 'Ċ'
0.01.031.304 I print_info: EOG token = 156892 '<|endoftext|>'
0.01.031.304 I print_info: EOG token = 156895 '<|role_end|>'
0.01.031.304 I print_info: max token length = 154
0.01.031.305 I load_tensors: loading model tensors, this can take a while... (load_mode = mlock)
0.02.804.333 I load_tensors: offloading output layer to GPU
0.02.804.338 I load_tensors: offloading 41 repeating layers to GPU
0.02.804.339 I load_tensors: offloaded 43/43 layers to GPU
0.02.804.346 I load_tensors: CPU model buffer size = 0.00 MiB
0.02.804.347 I load_tensors: CUDA0 model buffer size = 6831.14 MiB
0.02.804.351 I load_tensors: CPU model buffer size = 48310.23 MiB
0.51.627.617 I cmn common_init_: added <|endoftext|> logit bias = -inf
0.51.628.585 I cmn common_init_: added <|role_end|> logit bias = -inf
0.51.762.166 I llama_context: constructing llama_context
0.52.029.466 I llama_context: setting backend sampler for seq_id 0 (n = 4)
0.52.029.486 I llama_context: n_seq_max = 1
0.52.029.486 I llama_context: n_ctx = 120064
0.52.029.487 I llama_context: n_ctx_seq = 120064
0.52.029.487 I llama_context: n_batch = 3008
0.52.029.487 I llama_context: n_ubatch = 3008
0.52.029.490 I llama_context: causal_attn = 1
0.52.033.765 I llama_context: flash_attn = enabled
0.52.033.766 I llama_context: kv_unified = true
0.52.033.780 I llama_context: freq_base = 6000000.0
0.52.033.783 I llama_context: freq_scale = 1
0.52.033.784 I llama_context: n_rs_seq = 0
0.52.033.784 I llama_context: n_outputs_max = 1
0.52.033.785 I llama_context: n_ctx_seq (120064) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
0.52.119.219 I llama_context: CUDA_Host output buffer size = 2.40 MiB
0.52.132.088 I llama_kv_cache: CUDA0 KV buffer size = 923.34 MiB
0.52.137.746 I llama_kv_cache: size = 923.34 MiB (120064 cells, 7 layers, 1/1 seqs), K (bf16): 923.34 MiB, V (bf16): 0.00 MiB
0.52.137.755 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 576
0.52.137.756 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 0
0.52.138.682 I llama_memory_recurrent: CUDA0 RS buffer size = 74.92 MiB
0.52.138.691 I llama_memory_recurrent: size = 74.92 MiB ( 1 cells, 42 layers, 1 seqs 0 rs_seq), R (f32): 4.92 MiB, S (f32): 70.00 MiB
0.52.138.701 I sched_reserve: reserving ...
0.52.161.354 I resolve_fused_ops: resolving fused Gated Delta Net support:
0.52.262.130 I resolve_fused_ops: fused Gated Delta Net (autoregressive) enabled
0.52.263.957 I resolve_fused_ops: fused Gated Delta Net (chunked) enabled
0.52.263.960 I resolve_fused_ops: resolving fused Lightning Indexer support:
0.52.265.460 I resolve_fused_ops: Lightning Indexer enabled
0.52.265.463 I resolve_fused_ops: resolving fused DeepSeek V4 HC support:
0.52.266.898 I resolve_fused_ops: fused DeepSeek V4 HC pre enabled
0.52.268.373 I resolve_fused_ops: fused DeepSeek V4 HC comb enabled
0.52.269.889 I resolve_fused_ops: fused DeepSeek V4 HC post enabled
0.52.465.166 I sched_reserve: CUDA0 compute buffer size = 1752.34 MiB
0.52.465.170 I sched_reserve: CUDA_Host compute buffer size = 747.64 MiB
0.52.465.171 I sched_reserve: graph nodes = 5669
0.52.465.187 I sched_reserve: graph splits = 116 (with bs=3008), 78 (with bs=1)
0.52.465.190 I sched_reserve: reserve took 326.49 ms, sched copies = 1
0.52.490.567 I cmn common_init_: warming up the model with an empty run - please wait ... (--no-warmup to disable)
1.13.268.561 I cmn common_conte: the context does not support partial sequence removal
1.13.283.826 I srv load_model: speculative decoding will use checkpoints
1.13.283.831 I srv load_model: initializing, n_slots = 1, n_ctx_slot = 120064, kv_unified = 'true'
1.13.286.153 I spec common_specu: no implementations specified for speculative decoding
1.13.286.161 I slot load_model: id 0 | task -1 | new slot, n_ctx = 120064
1.13.296.273 I srv load_model: prompt cache is enabled, size limit: 1024 MiB
1.13.296.281 I srv load_model: use `--cache-ram 0` to disable the prompt cache
1.13.296.282 I srv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
1.13.296.283 I srv load_model: context checkpoints enabled, max = 32, min spacing = 8192
1.13.310.516 I srv init: idle slots will be saved to prompt cache and cleared upon starting a new task
1.13.515.846 I srv init: init: chat template, example_format: '<role>SYSTEM</role>You are a helpful assistant
detailed thinking on<|role_end|><role>HUMAN</role>Hello<|role_end|><role>ASSISTANT</role>
<think></think>Hi there<|role_end|><role>HUMAN</role>How are you?<|role_end|><role>ASSISTANT</role>
<think>'
1.13.538.501 I srv init: init: chat template, thinking = 1
1.13.567.405 I srv llama_server: model loaded
1.13.567.417 I srv llama_server: listening on http://127.0.0.1:8080
1.13.567.417 W srv llama_server: NOTICE: server default port will be changed to :9931 in a future release
1.13.567.417 W srv llama_server: ref: https://github.com/ggml-org/llama.cpp/pull/26508
1.13.574.165 I srv update_slots: all slots are idle
1.50.907.477 I srv server_strea: conv_id= (empty=1)
1.51.531.318 I srv operator(): chat format: peg-native
1.51.547.703 I slot get_availabl: id 0 | task -1 | - skipping, slot is empty
1.51.547.709 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
1.51.547.719 I srv get_availabl: updating prompt cache
1.51.548.589 I srv load: - looking for better prompt, base f_keep = -1.000, f_sim = 0.000
1.51.548.594 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 1024.000 MiB, 120064 tokens, 1073741824 est)
1.51.548.596 I srv get_availabl: prompt cache update took 0.87 ms
1.51.576.489 I cmn common_reaso: activated, budget=32000 tokens
1.51.576.502 W common_sampler_init: backend sampling is not compatible with grammar, disabling
1.51.577.067 I slot launch_slot_: id 0 | task -1 | sampler chain: logits -> top-k -> top-p -> temp-ext -> dist
1.51.577.090 I slot launch_slot_: id 0 | task -1 | sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 120064
top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.600
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
1.51.577.093 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
1.51.577.746 I slot operator(): id 0 | task 0 | new prompt, n_ctx_slot = 120064, n_keep = 0, task.n_tokens = 75903
1.51.578.707 I slot operator(): id 0 | task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
1.51.592.963 I sched_reserve: reserving ...
1.51.769.011 I sched_reserve: CUDA0 compute buffer size = 1752.34 MiB
1.51.769.015 I sched_reserve: CUDA_Host compute buffer size = 747.64 MiB
1.51.769.016 I sched_reserve: graph nodes = 5632
1.51.769.017 I sched_reserve: graph splits = 116 (with bs=3008), 78 (with bs=1)
1.51.769.020 I sched_reserve: reserve took 176.04 ms, sched copies = 1
1.57.351.622 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 3008, progress = 0.04, t = 5.77 s / 520.97 tokens per second
1.57.351.626 I slot operator(): id 0 | task 0 | cached n_tokens = 3008, memory_seq_rm [3008, end)
2.02.559.247 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 6016, progress = 0.08, t = 10.98 s / 547.83 tokens per second
2.02.559.251 I slot operator(): id 0 | task 0 | cached n_tokens = 6016, memory_seq_rm [6016, end)
2.07.244.385 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 9024, progress = 0.12, t = 15.67 s / 576.00 tokens per second
2.07.244.389 I slot operator(): id 0 | task 0 | cached n_tokens = 9024, memory_seq_rm [9024, end)
2.11.305.192 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 12032, progress = 0.16, t = 19.73 s / 609.91 tokens per second
2.11.305.196 I slot operator(): id 0 | task 0 | cached n_tokens = 12032, memory_seq_rm [12032, end)
2.15.845.162 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 15040, progress = 0.20, t = 24.27 s / 619.76 tokens per second
2.15.845.166 I slot operator(): id 0 | task 0 | cached n_tokens = 15040, memory_seq_rm [15040, end)
2.20.456.507 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 18048, progress = 0.24, t = 28.88 s / 624.96 tokens per second
2.20.456.511 I slot operator(): id 0 | task 0 | cached n_tokens = 18048, memory_seq_rm [18048, end)
2.25.206.641 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 21056, progress = 0.28, t = 33.63 s / 626.13 tokens per second
2.25.206.644 I slot operator(): id 0 | task 0 | cached n_tokens = 21056, memory_seq_rm [21056, end)
2.30.430.391 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 24064, progress = 0.32, t = 38.85 s / 619.37 tokens per second
2.30.430.397 I slot operator(): id 0 | task 0 | cached n_tokens = 24064, memory_seq_rm [24064, end)
2.35.444.468 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 27072, progress = 0.36, t = 43.87 s / 617.14 tokens per second
2.35.444.475 I slot operator(): id 0 | task 0 | cached n_tokens = 27072, memory_seq_rm [27072, end)
2.40.841.838 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 30080, progress = 0.40, t = 49.26 s / 610.59 tokens per second
2.40.841.843 I slot operator(): id 0 | task 0 | cached n_tokens = 30080, memory_seq_rm [30080, end)
2.43.609.589 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 30445, progress = 0.40, t = 52.03 s / 585.12 tokens per second
2.43.609.593 I slot operator(): id 0 | task 0 | cached n_tokens = 30445, memory_seq_rm [30445, end)
2.43.670.143 I slot create_check: id 0 | task 0 | created context checkpoint 1 of 32 (pos_min = 30444, pos_max = 30444, n_tokens = 30445, size = 74.923 MiB)
2.49.561.167 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 33453, progress = 0.44, t = 57.98 s / 576.94 tokens per second
2.49.561.172 I slot operator(): id 0 | task 0 | cached n_tokens = 33453, memory_seq_rm [33453, end)
2.55.434.005 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 36461, progress = 0.48, t = 63.86 s / 570.99 tokens per second
2.55.434.009 I slot operator(): id 0 | task 0 | cached n_tokens = 36461, memory_seq_rm [36461, end)
3.01.504.348 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 39469, progress = 0.52, t = 69.93 s / 564.43 tokens per second
3.01.504.355 I slot operator(): id 0 | task 0 | cached n_tokens = 39469, memory_seq_rm [39469, end)
3.07.719.944 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 42477, progress = 0.56, t = 76.14 s / 557.86 tokens per second
3.07.719.952 I slot operator(): id 0 | task 0 | cached n_tokens = 42477, memory_seq_rm [42477, end)
3.14.285.187 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 45485, progress = 0.60, t = 82.71 s / 549.95 tokens per second
3.14.285.193 I slot operator(): id 0 | task 0 | cached n_tokens = 45485, memory_seq_rm [45485, end)
3.20.916.378 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 48493, progress = 0.64, t = 89.34 s / 542.80 tokens per second
3.20.916.382 I slot operator(): id 0 | task 0 | cached n_tokens = 48493, memory_seq_rm [48493, end)
3.27.813.095 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 51501, progress = 0.68, t = 96.24 s / 535.16 tokens per second
3.27.813.100 I slot operator(): id 0 | task 0 | cached n_tokens = 51501, memory_seq_rm [51501, end)
3.34.398.132 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 54509, progress = 0.72, t = 102.82 s / 530.14 tokens per second
3.34.398.140 I slot operator(): id 0 | task 0 | cached n_tokens = 54509, memory_seq_rm [54509, end)
3.41.413.816 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 57517, progress = 0.76, t = 109.84 s / 523.66 tokens per second
3.41.413.822 I slot operator(): id 0 | task 0 | cached n_tokens = 57517, memory_seq_rm [57517, end)
3.48.076.256 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 60525, progress = 0.80, t = 116.50 s / 519.53 tokens per second
3.48.076.262 I slot operator(): id 0 | task 0 | cached n_tokens = 60525, memory_seq_rm [60525, end)
3.55.211.865 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 63533, progress = 0.84, t = 123.63 s / 513.88 tokens per second
3.55.211.871 I slot operator(): id 0 | task 0 | cached n_tokens = 63533, memory_seq_rm [63533, end)
4.02.576.015 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 66541, progress = 0.88, t = 131.00 s / 507.95 tokens per second
4.02.576.032 I slot operator(): id 0 | task 0 | cached n_tokens = 66541, memory_seq_rm [66541, end)
4.08.194.217 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 67918, progress = 0.89, t = 136.62 s / 497.14 tokens per second
4.08.194.221 I slot operator(): id 0 | task 0 | cached n_tokens = 67918, memory_seq_rm [67918, end)
4.08.355.217 I slot create_check: id 0 | task 0 | created context checkpoint 2 of 32 (pos_min = 67917, pos_max = 67917, n_tokens = 67918, size = 74.923 MiB)
4.13.550.642 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 69558, progress = 0.92, t = 141.97 s / 489.94 tokens per second
4.13.550.645 I slot operator(): id 0 | task 0 | cached n_tokens = 69558, memory_seq_rm [69558, end)
4.20.660.842 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 72566, progress = 0.96, t = 149.08 s / 486.75 tokens per second
4.20.660.846 I slot operator(): id 0 | task 0 | cached n_tokens = 72566, memory_seq_rm [72566, end)
4.23.727.090 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 72895, progress = 0.96, t = 152.15 s / 479.10 tokens per second
4.23.727.094 I slot operator(): id 0 | task 0 | cached n_tokens = 72895, memory_seq_rm [72895, end)
4.23.778.933 I slot create_check: id 0 | task 0 | created context checkpoint 3 of 32 (pos_min = 72894, pos_max = 72894, n_tokens = 72895, size = 74.923 MiB)
4.30.911.031 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 75890, progress = 1.00, t = 159.33 s / 476.30 tokens per second
4.30.911.035 I slot operator(): id 0 | task 0 | cached n_tokens = 75890, memory_seq_rm [75890, end)
4.31.293.884 I slot create_check: id 0 | task 0 | created context checkpoint 4 of 32 (pos_min = 75889, pos_max = 75889, n_tokens = 75890, size = 74.923 MiB)
4.31.798.962 I slot print_timing: id 0 | task 0 | prompt processing, n_tokens = 75899, progress = 1.00, t = 160.22 s / 473.71 tokens per second
4.31.798.966 I slot operator(): id 0 | task 0 | cached n_tokens = 75899, memory_seq_rm [75899, end)
4.31.819.267 I slot init_sampler: id 0 | task 0 | init sampler, took 8.09 ms, tokens: text = 75903, total = 75903
4.31.834.242 I slot create_check: id 0 | task 0 | created context checkpoint 5 of 32 (pos_min = 75898, pos_max = 75898, n_tokens = 75899, size = 74.923 MiB)
4.39.068.259 I slot print_timing: id 0 | task 0 | n_decoded = 100, tg = 14.36 t/s, tg_3s = 14.36 t/s
4.42.105.103 I slot print_timing: id 0 | task 0 | n_decoded = 141, tg = 14.10 t/s, tg_3s = 13.50 t/s
4.45.144.858 I slot print_timing: id 0 | task 0 | n_decoded = 187, tg = 14.34 t/s, tg_3s = 15.13 t/s
4.48.155.182 I slot print_timing: id 0 | task 0 | n_decoded = 232, tg = 14.45 t/s, tg_3s = 14.95 t/s
4.49.868.631 W srv stop: cancel task, id_task = 0
4.49.888.188 I slot release: id 0 | task 0 | stop processing: n_tokens = 76159, truncated = 0
4.49.888.207 I srv update_slots: all slots are idle
4.58.638.030 I srv server_strea: conv_id= (empty=1)
4.58.820.135 I srv operator(): chat format: peg-native
4.58.828.895 I slot get_availabl: id 0 | task -1 | - checking sim = 1.000 (75897/75918) > 0.100
4.58.828.900 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 1.000 (> 0.100 thold), f_keep = 0.997
4.58.832.727 I cmn common_reaso: activated, budget=32000 tokens
4.58.832.738 W common_sampler_init: backend sampling is not compatible with grammar, disabling
4.58.833.415 I slot launch_slot_: id 0 | task -1 | sampler chain: logits -> top-k -> top-p -> temp-ext -> dist
4.58.833.452 I slot launch_slot_: id 0 | task -1 | sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 120064
top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.600
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
4.58.833.455 I slot launch_slot_: id 0 | task 288 | processing task, is_child = 0
4.58.833.462 I slot operator(): id 0 | task 288 | new prompt, n_ctx_slot = 120064, n_keep = 0, task.n_tokens = 75918
4.58.833.513 I slot operator(): id 0 | task 288 | checking checkpoint with [75898, 75898] against 75897...
4.58.833.514 I slot operator(): id 0 | task 288 | checking checkpoint with [75889, 75889] against 75897...
4.58.845.816 I slot operator(): id 0 | task 288 | restored context checkpoint (pos_min = 75889, pos_max = 75889, n_tokens = 75890, n_past = 75890, size = 74.923 MiB)
4.58.845.821 I slot operator(): id 0 | task 288 | erased invalidated context checkpoint (pos_min = 75898, pos_max = 75898, n_tokens = 75899, n_swa = 0, pos_next = 75890, size = 74.923 MiB)
4.58.847.759 I slot operator(): id 0 | task 288 | cached n_tokens = 75890, memory_seq_rm [75890, end)
4.58.848.105 I slot create_check: id 0 | task 288 | erasing context checkpoint too close to an earlier one (pos_min = 72894, pos_max = 72894, n_tokens = 72895, size = 74.923 MiB)
4.58.854.349 I slot create_check: id 0 | task 288 | erasing context checkpoint too close to an earlier one (pos_min = 75889, pos_max = 75889, n_tokens = 75890, size = 74.923 MiB)
4.58.873.336 I slot create_check: id 0 | task 288 | created context checkpoint 3 of 32 (pos_min = 75889, pos_max = 75889, n_tokens = 75890, size = 74.923 MiB)
4.59.247.232 I slot operator(): id 0 | task 288 | cached n_tokens = 75896, memory_seq_rm [75896, end)
4.59.267.573 I slot create_check: id 0 | task 288 | created context checkpoint 4 of 32 (pos_min = 75895, pos_max = 75895, n_tokens = 75896, size = 74.923 MiB)
5.00.345.694 I slot operator(): id 0 | task 288 | cached n_tokens = 75914, memory_seq_rm [75914, end)
5.00.353.940 I slot init_sampler: id 0 | task 288 | init sampler, took 8.14 ms, tokens: text = 75918, total = 75918
5.00.368.968 I slot create_check: id 0 | task 288 | created context checkpoint 5 of 32 (pos_min = 75913, pos_max = 75913, n_tokens = 75914, size = 74.923 MiB)
5.07.264.198 I slot print_timing: id 0 | task 288 | n_decoded = 100, tg = 14.91 t/s, tg_3s = 14.91 t/s
5.07.459.876 I cmn common_reaso: deactivated (natural end)
5.10.321.943 I slot print_timing: id 0 | task 288 | n_decoded = 144, tg = 14.75 t/s, tg_3s = 14.39 t/s
5.13.371.614 I slot print_timing: id 0 | task 288 | n_decoded = 189, tg = 14.75 t/s, tg_3s = 14.76 t/s
5.16.408.689 I slot print_timing: id 0 | task 288 | n_decoded = 234, tg = 14.76 t/s, tg_3s = 14.82 t/s
5.19.414.881 I slot print_timing: id 0 | task 288 | n_decoded = 279, tg = 14.80 t/s, tg_3s = 14.97 t/s
5.22.463.704 I slot print_timing: id 0 | task 288 | n_decoded = 324, tg = 14.79 t/s, tg_3s = 14.76 t/s
5.25.481.639 I slot print_timing: id 0 | task 288 | n_decoded = 369, tg = 14.81 t/s, tg_3s = 14.91 t/s
5.28.501.634 I slot print_timing: id 0 | task 288 | n_decoded = 413, tg = 14.78 t/s, tg_3s = 14.57 t/s
5.31.508.425 I slot print_timing: id 0 | task 288 | n_decoded = 458, tg = 14.80 t/s, tg_3s = 14.97 t/s
5.34.518.491 I slot print_timing: id 0 | task 288 | n_decoded = 503, tg = 14.81 t/s, tg_3s = 14.95 t/s
5.37.548.419 I slot print_timing: id 0 | task 288 | n_decoded = 547, tg = 14.79 t/s, tg_3s = 14.52 t/s
5.40.612.162 I slot print_timing: id 0 | task 288 | n_decoded = 592, tg = 14.78 t/s, tg_3s = 14.69 t/s
5.43.612.246 I slot print_timing: id 0 | task 288 | n_decoded = 635, tg = 14.75 t/s, tg_3s = 14.33 t/s
5.46.625.991 I slot print_timing: id 0 | task 288 | n_decoded = 680, tg = 14.76 t/s, tg_3s = 14.93 t/s
5.49.687.849 I slot print_timing: id 0 | task 288 | n_decoded = 727, tg = 14.80 t/s, tg_3s = 15.35 t/s
5.52.712.670 I slot print_timing: id 0 | task 288 | n_decoded = 773, tg = 14.82 t/s, tg_3s = 15.21 t/s
5.55.740.121 I slot print_timing: id 0 | task 288 | n_decoded = 820, tg = 14.86 t/s, tg_3s = 15.52 t/s
5.58.795.374 I slot print_timing: id 0 | task 288 | n_decoded = 866, tg = 14.87 t/s, tg_3s = 15.06 t/s
6.01.842.891 I slot print_timing: id 0 | task 288 | n_decoded = 912, tg = 14.88 t/s, tg_3s = 15.09 t/s
6.04.851.576 I slot print_timing: id 0 | task 288 | n_decoded = 959, tg = 14.92 t/s, tg_3s = 15.62 t/s
6.07.913.295 I slot print_timing: id 0 | task 288 | n_decoded = 1006, tg = 14.94 t/s, tg_3s = 15.35 t/s
6.10.970.907 I slot print_timing: id 0 | task 288 | n_decoded = 1053, tg = 14.95 t/s, tg_3s = 15.37 t/s
6.13.994.481 I slot print_timing: id 0 | task 288 | n_decoded = 1099, tg = 14.97 t/s, tg_3s = 15.21 t/s
6.16.999.322 I slot print_timing: id 0 | task 288 | n_decoded = 1145, tg = 14.98 t/s, tg_3s = 15.31 t/s
6.20.000.857 I slot print_timing: id 0 | task 288 | n_decoded = 1191, tg = 14.99 t/s, tg_3s = 15.33 t/s
6.23.009.168 I slot print_timing: id 0 | task 288 | n_decoded = 1237, tg = 15.00 t/s, tg_3s = 15.29 t/s
6.26.062.595 I slot print_timing: id 0 | task 288 | n_decoded = 1284, tg = 15.02 t/s, tg_3s = 15.39 t/s
6.29.123.967 I slot print_timing: id 0 | task 288 | n_decoded = 1331, tg = 15.03 t/s, tg_3s = 15.35 t/s
6.32.177.905 I slot print_timing: id 0 | task 288 | n_decoded = 1378, tg = 15.04 t/s, tg_3s = 15.39 t/s
6.35.206.323 I slot print_timing: id 0 | task 288 | n_decoded = 1424, tg = 15.05 t/s, tg_3s = 15.19 t/s
6.38.228.636 I slot print_timing: id 0 | task 288 | n_decoded = 1470, tg = 15.05 t/s, tg_3s = 15.22 t/s
6.41.234.895 I slot print_timing: id 0 | task 288 | n_decoded = 1517, tg = 15.07 t/s, tg_3s = 15.63 t/s
6.44.241.152 I slot print_timing: id 0 | task 288 | n_decoded = 1563, tg = 15.07 t/s, tg_3s = 15.30 t/s
6.47.284.318 I slot print_timing: id 0 | task 288 | n_decoded = 1610, tg = 15.09 t/s, tg_3s = 15.44 t/s
6.50.308.636 I slot print_timing: id 0 | task 288 | n_decoded = 1656, tg = 15.09 t/s, tg_3s = 15.21 t/s
6.53.310.106 I slot print_timing: id 0 | task 288 | n_decoded = 1702, tg = 15.10 t/s, tg_3s = 15.33 t/s
6.56.348.836 I slot print_timing: id 0 | task 288 | n_decoded = 1749, tg = 15.11 t/s, tg_3s = 15.47 t/s
6.59.374.086 I slot print_timing: id 0 | task 288 | n_decoded = 1795, tg = 15.11 t/s, tg_3s = 15.21 t/s
7.02.432.256 I slot print_timing: id 0 | task 288 | n_decoded = 1842, tg = 15.11 t/s, tg_3s = 15.37 t/s
7.03.663.676 I slot print_timing: id 0 | task 288 | prompt eval time = 1725.19 ms / 28 tokens ( 61.61 ms per token, 16.23 tokens per second)
7.03.663.680 I slot print_timing: id 0 | task 288 | eval time = 123105.01 ms / 1861 tokens ( 66.15 ms per token, 15.12 tokens per second)
7.03.663.682 I slot print_timing: id 0 | task 288 | total time = 124830.20 ms / 1889 tokens
7.03.663.682 I slot print_timing: id 0 | task 288 | graphs reused = 2107
7.03.830.453 I slot release: id 0 | task 288 | stop processing: n_tokens = 77778, truncated = 0
7.03.831.570 I srv update_slots: all slots are idle
7.10.150.598 I srv server_strea: conv_id= (empty=1)
7.10.314.236 I srv operator(): chat format: peg-native
7.10.322.786 I slot get_availabl: id 0 | task -1 | - checking sim = 0.973 (77778/79902) > 0.100
7.10.322.792 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.973 (> 0.100 thold), f_keep = 1.000
7.10.326.691 I cmn common_reaso: activated, budget=32000 tokens
7.10.326.697 W common_sampler_init: backend sampling is not compatible with grammar, disabling
7.10.326.979 I slot launch_slot_: id 0 | task -1 | sampler chain: logits -> top-k -> top-p -> temp-ext -> dist
7.10.326.993 I slot launch_slot_: id 0 | task -1 | sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 120064
top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.600
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
7.10.326.995 I slot launch_slot_: id 0 | task 2152 | processing task, is_child = 0
7.10.327.003 I slot operator(): id 0 | task 2152 | new prompt, n_ctx_slot = 120064, n_keep = 0, task.n_tokens = 79902
7.10.327.046 I slot operator(): id 0 | task 2152 | cached n_tokens = 77778, memory_seq_rm [77778, end)
7.10.327.273 I slot create_check: id 0 | task 2152 | erasing context checkpoint too close to an earlier one (pos_min = 75889, pos_max = 75889, n_tokens = 75890, size = 74.923 MiB)
7.10.329.024 I slot create_check: id 0 | task 2152 | erasing context checkpoint too close to an earlier one (pos_min = 75895, pos_max = 75895, n_tokens = 75896, size = 74.923 MiB)
7.10.330.659 I slot create_check: id 0 | task 2152 | erasing context checkpoint too close to an earlier one (pos_min = 75913, pos_max = 75913, n_tokens = 75914, size = 74.923 MiB)
7.10.350.861 I slot create_check: id 0 | task 2152 | created context checkpoint 3 of 32 (pos_min = 77777, pos_max = 77777, n_tokens = 77778, size = 74.923 MiB)
7.15.798.933 I slot print_timing: id 0 | task 2152 | prompt processing, n_tokens = 1939, progress = 1.00, t = 5.47 s / 354.35 tokens per second
7.15.798.939 I slot operator(): id 0 | task 2152 | cached n_tokens = 79717, memory_seq_rm [79717, end)
7.16.048.417 I slot create_check: id 0 | task 2152 | created context checkpoint 4 of 32 (pos_min = 79716, pos_max = 79716, n_tokens = 79717, size = 74.923 MiB)
7.18.723.977 I slot print_timing: id 0 | task 2152 | prompt processing, n_tokens = 2120, progress = 1.00, t = 8.40 s / 252.47 tokens per second
7.18.723.981 I slot operator(): id 0 | task 2152 | cached n_tokens = 79898, memory_seq_rm [79898, end)
7.18.732.882 I slot init_sampler: id 0 | task 2152 | init sampler, took 8.74 ms, tokens: text = 79902, total = 79902
7.18.761.969 I slot create_check: id 0 | task 2152 | created context checkpoint 5 of 32 (pos_min = 79897, pos_max = 79897, n_tokens = 79898, size = 74.923 MiB)
7.22.703.046 I cmn common_reaso: deactivated (natural end)
7.25.752.371 I slot print_timing: id 0 | task 2152 | n_decoded = 100, tg = 14.73 t/s, tg_3s = 14.73 t/s
7.26.160.562 I slot print_timing: id 0 | task 2152 | prompt eval time = 8637.02 ms / 2124 tokens ( 4.07 ms per token, 245.92 tokens per second)
7.26.160.566 I slot print_timing: id 0 | task 2152 | eval time = 7196.53 ms / 106 tokens ( 67.89 ms per token, 14.73 tokens per second)
7.26.160.567 I slot print_timing: id 0 | task 2152 | total time = 15833.55 ms / 2230 tokens
7.26.160.568 I slot print_timing: id 0 | task 2152 | graphs reused = 2211
7.26.163.252 I slot release: id 0 | task 2152 | stop processing: n_tokens = 80007, truncated = 0
7.26.163.614 I srv update_slots: all slots are idle
7.26.570.292 I srv server_strea: conv_id= (empty=1)
7.26.782.355 I srv operator(): chat format: peg-native
7.26.793.796 I slot get_availabl: id 0 | task -1 | - checking sim = 0.996 (79718/80077) > 0.100
7.26.793.801 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.996 (> 0.100 thold), f_keep = 0.996
7.26.797.518 I cmn common_reaso: activated, budget=32000 tokens
7.26.797.526 W common_sampler_init: backend sampling is not compatible with grammar, disabling
7.26.797.757 I slot launch_slot_: id 0 | task -1 | sampler chain: logits -> top-k -> top-p -> temp-ext -> dist
7.26.797.773 I slot launch_slot_: id 0 | task -1 | sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 120064
top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.600
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
7.26.797.776 I slot launch_slot_: id 0 | task 2261 | processing task, is_child = 0
7.26.797.786 I slot operator(): id 0 | task 2261 | new prompt, n_ctx_slot = 120064, n_keep = 0, task.n_tokens = 80077
7.26.797.827 I slot operator(): id 0 | task 2261 | checking checkpoint with [79897, 79897] against 79718...
7.26.797.829 I slot operator(): id 0 | task 2261 | checking checkpoint with [79716, 79716] against 79718...
7.26.809.492 I slot operator(): id 0 | task 2261 | restored context checkpoint (pos_min = 79716, pos_max = 79716, n_tokens = 79717, n_past = 79717, size = 74.923 MiB)
7.26.809.497 I slot operator(): id 0 | task 2261 | erased invalidated context checkpoint (pos_min = 79897, pos_max = 79897, n_tokens = 79898, n_swa = 0, pos_next = 79717, size = 74.923 MiB)
7.26.811.350 I slot operator(): id 0 | task 2261 | cached n_tokens = 79717, memory_seq_rm [79717, end)
7.26.811.617 I slot create_check: id 0 | task 2261 | erasing context checkpoint too close to an earlier one (pos_min = 79716, pos_max = 79716, n_tokens = 79717, size = 74.923 MiB)
7.26.830.780 I slot create_check: id 0 | task 2261 | created context checkpoint 4 of 32 (pos_min = 79716, pos_max = 79716, n_tokens = 79717, size = 74.923 MiB)
7.30.056.837 I slot print_timing: id 0 | task 2261 | prompt processing, n_tokens = 258, progress = 1.00, t = 3.26 s / 79.16 tokens per second
7.30.056.842 I slot operator(): id 0 | task 2261 | cached n_tokens = 79975, memory_seq_rm [79975, end)
7.30.105.160 I slot create_check: id 0 | task 2261 | created context checkpoint 5 of 32 (pos_min = 79974, pos_max = 79974, n_tokens = 79975, size = 74.923 MiB)
7.32.323.165 I slot print_timing: id 0 | task 2261 | prompt processing, n_tokens = 356, progress = 1.00, t = 5.53 s / 64.43 tokens per second
7.32.323.169 I slot operator(): id 0 | task 2261 | cached n_tokens = 80073, memory_seq_rm [80073, end)
7.32.334.016 I slot init_sampler: id 0 | task 2261 | init sampler, took 10.75 ms, tokens: text = 80077, total = 80077
7.32.353.908 I slot create_check: id 0 | task 2261 | created context checkpoint 6 of 32 (pos_min = 80072, pos_max = 80072, n_tokens = 80073, size = 74.923 MiB)
7.36.423.851 I cmn common_reaso: deactivated (natural end)
7.39.084.058 I slot print_timing: id 0 | task 2261 | n_decoded = 100, tg = 15.30 t/s, tg_3s = 15.30 t/s
7.42.106.177 I slot print_timing: id 0 | task 2261 | n_decoded = 146, tg = 15.27 t/s, tg_3s = 15.22 t/s
7.45.121.760 I slot print_timing: id 0 | task 2261 | n_decoded = 190, tg = 15.11 t/s, tg_3s = 14.59 t/s
7.48.156.417 I slot print_timing: id 0 | task 2261 | n_decoded = 237, tg = 15.18 t/s, tg_3s = 15.49 t/s
7.51.187.141 I slot print_timing: id 0 | task 2261 | n_decoded = 284, tg = 15.24 t/s, tg_3s = 15.51 t/s
7.54.217.409 I slot print_timing: id 0 | task 2261 | n_decoded = 328, tg = 15.14 t/s, tg_3s = 14.52 t/s
7.57.241.288 I slot print_timing: id 0 | task 2261 | n_decoded = 373, tg = 15.10 t/s, tg_3s = 14.88 t/s
8.00.299.139 I slot print_timing: id 0 | task 2261 | n_decoded = 419, tg = 15.10 t/s, tg_3s = 15.04 t/s
8.03.334.834 I slot print_timing: id 0 | task 2261 | n_decoded = 465, tg = 15.10 t/s, tg_3s = 15.15 t/s
8.06.343.865 I slot print_timing: id 0 | task 2261 | n_decoded = 509, tg = 15.06 t/s, tg_3s = 14.62 t/s
8.09.355.377 I slot print_timing: id 0 | task 2261 | prompt eval time = 5749.32 ms / 360 tokens ( 15.97 ms per token, 62.62 tokens per second)
8.09.355.382 I slot print_timing: id 0 | task 2261 | eval time = 36808.26 ms / 553 tokens ( 66.56 ms per token, 15.02 tokens per second)
8.09.355.384 I slot print_timing: id 0 | task 2261 | total time = 42557.58 ms / 913 tokens
8.09.355.385 I slot print_timing: id 0 | task 2261 | graphs reused = 2760
8.09.359.280 I slot release: id 0 | task 2261 | stop processing: n_tokens = 80629, truncated = 0
8.09.359.724 I srv update_slots: all slots are idle
8.12.918.586 I srv server_strea: conv_id= (empty=1)
8.13.079.883 I srv operator(): chat format: peg-native
8.13.088.640 I slot get_availabl: id 0 | task -1 | - checking sim = 0.991 (79976/80687) > 0.100
8.13.088.645 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.991 (> 0.100 thold), f_keep = 0.992
8.13.092.811 I cmn common_reaso: activated, budget=32000 tokens
8.13.092.818 W common_sampler_init: backend sampling is not compatible with grammar, disabling
8.13.093.118 I slot launch_slot_: id 0 | task -1 | sampler chain: logits -> top-k -> top-p -> temp-ext -> dist
8.13.093.130 I slot launch_slot_: id 0 | task -1 | sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 120064
top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.600
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
8.13.093.158 I slot launch_slot_: id 0 | task 2817 | processing task, is_child = 0
8.13.093.167 I slot operator(): id 0 | task 2817 | new prompt, n_ctx_slot = 120064, n_keep = 0, task.n_tokens = 80687
8.13.093.199 I slot operator(): id 0 | task 2817 | reusing chunk with size 558, shifting KV cache [80071, 80629) -> [79976, 80534)
8.13.093.915 I slot operator(): id 0 | task 2817 | cached n_tokens = 80534, memory_seq_rm [80534, end)
8.13.094.006 I slot create_check: id 0 | task 2817 | erasing context checkpoint too close to an earlier one (pos_min = 79716, pos_max = 79716, n_tokens = 79717, size = 74.923 MiB)
8.13.096.051 I slot create_check: id 0 | task 2817 | erasing context checkpoint too close to an earlier one (pos_min = 79974, pos_max = 79974, n_tokens = 79975, size = 74.923 MiB)
8.13.098.036 I slot create_check: id 0 | task 2817 | erasing context checkpoint too close to an earlier one (pos_min = 80072, pos_max = 80072, n_tokens = 80073, size = 74.923 MiB)
8.13.124.148 I slot create_check: id 0 | task 2817 | created context checkpoint 4 of 32 (pos_min = 80533, pos_max = 80533, n_tokens = 80534, size = 74.923 MiB)
cmd_child_to_router:error:G:\xlam2\atomic-llama-cpp-turboquant\ggml\src\ggml-cuda\rope.cu:531: GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16) failed
Error running server: Command '['.\\llama-server.exe', '-m', 'D:\\Ling-3.0-flash-AD-IQ3_S-00001-of-00002.gguf', '--reasoning-preserve', '--reasoning-budget', '32000', '--reasoning-budget-message', "I've thought enough. Answering now with what I have.", '--fit', 'on', '--n-cpu-moe', '40', '-bs', '-lm', 'mlock', '-c', '120000', '--cache-type-k', 'bf16', '--cache-type-v', 'bf16', '-np', '1', '-fa', 'on', '-t', '8', '-tb', '8', '-b', '3008', '-ub', '3008', '--jinja', '-kvu', '--temp', '0.6', '--top-p', '0.95', '--top-k', '20', '--samplers', 'top_k;top_p;temperature', '--alias', 'laguna-s-2.1', '--cache-reuse', '256', '--cache-ram', '1024', '--host', '127.0.0.1', '--port', '8080', '--verbosity', '4', '--chat-template-file', 'G:\\xlam3\\chat_template_ling.jinja']' returned non-zero exit status 3221226505.
Name and Version
.\llama-server.exe --version
version: b10269-1.5.1 (build 10696, commit cd56093)
built with Clang 22.1.8 for Windows AMD64
Operating systems
Windows
GGML backends
CUDA
Hardware
Ryzen 7 5700x3D + RTX 5070 + 56 gb ram
Models
Ling-3.0-flash-AD-IQ3_S-00001-of-00002.gguf
Problem description & steps to reproduce
At some point during generation it crashes with cmd_child_to_router:error:G:\xlam2\atomic-llama-cpp-turboquant\ggml\src\ggml-cuda\rope.cu:531: GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16) failed randomly.
Running with:
cat '.!start - ling.py'
import os
import subprocess
import sys
env = os.environ.copy()
cmd = [
r".\llama-server.exe",
"-m", r"D:\Ling-3.0-flash-AD-IQ3_S-00001-of-00002.gguf",
"--reasoning-preserve",
"--reasoning-budget", "32000",
"--reasoning-budget-message", "I've thought enough. Answering now with what I have.",
"--fit", "on",
"--n-cpu-moe", "40", "-bs",
"-lm", "mlock",
"-c", "120000",
"--cache-type-k", "bf16",
"--cache-type-v", "bf16",
"-np", "1",
"-fa", "on",
"-t", "8",
"-tb", "8",
"-b", "3008",
"-ub", "3008",
"--jinja",
"-kvu",
"--temp", "0.6",
"--top-p", "0.95",
"--top-k", "20",
"--samplers", "top_k;top_p;temperature",
"--alias", "ling-flash-3.0",
"--cache-reuse", "256",
"--cache-ram", "1024",
"--host", "127.0.0.1",
"--port", "8080",
"--verbosity", "4",
"--chat-template-file", r"G:\xlam3\chat_template_ling.jinja",
]
if name == "main":
try:
print("Starting llama-server...")
subprocess.run(cmd, env=env, check=True)
except KeyboardInterrupt:
print("\nServer stopped by user.")
except Exception as e:
print(f"\nError running server: {e}")
#removing --cache-reuse avoids it
First Bad Commit
No response
Relevant log output
Logs