Display the provider/engine being used to run the model

[benoitf]

Today if I launch the command on my mac with the command:

ramalama run tinyllama

if my podman machine is started, it'll deploy it to podman and it it's stopped it'll use llama-run. So depending on when I'm running the exact same instruction, it's not using the same backend.

I should prefix all my commands with

RAMALAMA_CONTAINER_ENGINE=podman ramalama run tinyllama

but it says 'container engine', so I'm not sure it's a reliable flag as well as I could image it would still be able to default to llama-run command

so IMHO it would be nice if I got a small prompt

$ ramalama run tinyllama
🦭 >

$ ramalama run tinyllama
🐋 >

$ ramalama run tinyllama
🐋 >

so I know it's using podman to run my model

or

$ ramalama run tinyllama
using podman (🦭) provider
>

or anything that could let me easily know where is running my model

(using --debug is producing way too much logs)

[rhatdan]

That would be cool, interested in opening a PR. Need to fail silently if graphic not available.

[ericcurtin]

Interface comes from llama.cpp's llama-run FWIW

[benoitf]

yes this is why I added the

or

$ ramalama run tinyllama
using podman (🦭) provider
>

where one line is added before the llama-run prompt

[rhatdan]

Couldn't we add a llama-run --prompt "🐋 >" ...

[benoitf]

not for now https://github.com/ggerganov/llama.cpp/blob/5b3cd1c0b90970a676ff27b1ad4d3d99a175c1b7/examples/run/run.cpp#L919

[ericcurtin]

Couldn't we add a llama-run --prompt "🐋 >" ...

Maybe, I'm not sure if the underlying library for this (linenoise.cpp) handles utf8/emoji. Of course it could be added 😊 But might not be as simple as it first seems.

[rhatdan]

Would be a nice feature though.

[benoitf]

Couldn't we add a llama-run --prompt "🐋 >" ...

Maybe, I'm not sure if the underlying library for this (linenoise.cpp) handles utf8/emoji. Of course it could be added 😊 But might not be as simple as it first seems.

it looks like it works if I replace hello in your forked repo https://github.com/ericcurtin/linenoise.cpp

[ericcurtin]

Ha cool, the reason I was sure, is there are several versions of this patch that are not merged:

antirez/linenoise#187

because linenoise.cpp is C++, converting everything to std::string might be enough, but you've already proved it works for the prompt part, so there might be enough enabled for the prompt feature of that library.