Getting exception: "llama_decode failed: 'NoKvSlot'" when LLM analyze text (news)

[MagicMaxxx]

Hello!

This exception happens not on a first prompt, but when I provided a list of news (in Russian, 10 rows with title, date and category) and then I ask LLM to analyze and write news related to sport topic. I've attached a list of news that I use in my prompt.
RbcRuNews.txt. The prompt usually looks like this: "вот список новостей, напиши только те что относятся к спорту. " + text from the file. Also, this exception not appears when I use a prompt about data stored in the LLM...

Stack trace (last rows):
at LLama.StatelessExecutor.d__9.MoveNext()
at LLama.StatelessExecutor.d__9.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
at LLama.LLamaTransforms.KeywordTextOutputStreamTransform.d__4.MoveNext()
at LLama.LLamaTransforms.KeywordTextOutputStreamTransform.d__4.MoveNext()
at LLama.LLamaTransforms.KeywordTextOutputStreamTransform.d__4.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
at LLamaSharp.SemanticKernel.ChatCompletion.LLamaSharpChatCompletion.d__11.MoveNext()
at LLamaSharp.SemanticKernel.ChatCompletion.LLamaSharpChatCompletion.d__11.MoveNext()
at LLamaSharp.SemanticKernel.ChatCompletion.LLamaSharpChatCompletion.d__11.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)

Multiple LLMs, got this exception on:

• TheBloke\llama-2-7b-chat.Q4_K_M.gguf
• TheBloke\mistral-7b-openorca.Q4_K_M.gguf
• IlyaGusev\saiga_mistral_7b_model-q4_K.gguf
• TheBloke\phi-2.Q5_K_S.gguf

LlamaSharp version is 0.10.1.

I'll update this ticket in the case if I'll get new details

[MagicMaxxx]

PROBABLY (!!) the reason of that exception is that that every news row started with "Дата:" (which is "Date:" in Russian) and LLM (?) decides that it's a kind of Role... So when I added minus before the Date ("- Дата:") then I can't reproduce the exception... Checking it...

[MagicMaxxx]

I've got this exception again, even if I used "minus" signs before the every news row.

In StatelessExecutor, in the line 81, it comes from LlamaContext:
DecodeResult decodeResult = await Context.DecodeAsync(_batch, cancellationToken);
if (decodeResult != 0)
{
throw new LLamaDecodeError(decodeResult);
}

[martindevans]

The NoKvSlot error means that there's not enough memory to hold all of the tokens you're trying to submit to the system.

You'll probably need to reduce the amount of text you are submitting in the initial prompt.

[MagicMaxxx]

ok, got it, thanks for the info!

I increased ModelParams.ContextSize 1024 -> 4096, checking it now (BTW - will it help? I've just started learning LLMs and LLamaSharp)... The prompt I used before had length of 2009 symbols, and if my assumption is right, it caused these exceptions

UPD - this is for llama-2-7b-chat.Q4_K_M, I can see that Mistral-based LLMs can have 8K as the context size, will check it later.

[martindevans]

There are 2 "contexts" here, which means different things. There's actually a PR in llama.cpp to change the naming, because it's so confusing!

There is a pre-allocated buffer (called the KV cache) which holds information about each token. That's what's set by the ModelParams.ContextSize parameter. If you had this set to 1024 then that will definitely cause an issue with trying to fit in 2009 tokens.

I can see that Mistral-based LLMs can have 8K as the context size

LLMs are trained for a certain number of positions (i.e. how many tokens they can understand, you can feed in more but this isn't something the system is trained for). That's what the 8k "context size" is here.

[martindevans]

I'll close this issue since it looks like it's been resolved :)