The AI suddenly stops the conversation

[toomy78]

Hello,

My goal would be to be able to provide a long text (about 10 Word pages) to the AI so that he can correct any errors in it for me.

However, when I provide my text, the AI starts correcting all the errors and then stops automatically.

Could you tell me what settings I'm using incorrectly or what I should do to improve them?

I use the Mixtral model.

Thanks a lot

[toomy78]

Hello,

Anyone have an answer please :) ?

Also, do you know if it's possible for several people to communicate simultaneously with the AI?

Thanks :)

[Jeximo]

There's not enough information.

I guess you're feeding your documents using the main example? How are you accessing llama.cpp? What settings do you use?

[toomy78]

I apologise for my very late reply.

I use the tool via the "server.exe" file using terminal commands.

In fact I noticed that the model was limited to 32,000 tokens.

My main aim would be for the tool/model to be able to give me an answer of around 100,000 words.

A very basic command : server.exe -m mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf -c 128000 -t 16

[Jeximo]

You might try a model trained with 128k tokens, like command r plus.

do you know if it's possible for several people to communicate simultaneously with the AI?

Yes, server readme shows -cb, --cont-batching enable continuous batching (default: disabled), and -np N, --parallel N: Set the number of slots for process requests (default: 1), so something like this:

./server ~/model.gguf -c 128000 -t 16 -cb -np 2

Note that the context size is divided between client slots(-np N), so -c 128000 -np 2 is 64k context per slot.

Here's related info.

[mscheong01]

Please provide the command, arguments and the prompt you used that could reproduce this issue.
One cause could be that the generation output exceeded the number of tokens to predict, and therefore stopped generating
https://github.com/ggerganov/llama.cpp/blob/d0e2f6416bd43eddb70137f6b96c7bc3d0246102/examples/infill/README.md?plain=1#L16

[toomy78]

I did not provide this information, so it should be -1