Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions tools/mtmd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,37 @@ Built upon `clip.cpp` (similar to `llava.cpp`), `libmtmd` offers several advanta
- **Improved UX/DX:** Features a more intuitive API, inspired by the `Processor` class in the Hugging Face `transformers` library.
- **Flexibility:** Designed to support multiple input types (text, audio, images) while respecting the wide variety of chat templates used by different models.

## Logging Configuration

By default, `libmtmd` logs messages directly to stderr. To integrate `libmtmd` logging with your application's logging system, you can use the `mtmd_log_set_llama_callback()` function to redirect all mtmd/clip logs through llama's logging callback.

**Example usage:**

```c
#include "llama.h"
#include "mtmd.h"

// Your custom logging callback
void my_log_callback(ggml_log_level level, const char * text, void * user_data) {
// Your logging logic here
printf("[%d] %s", level, text);
}

int main() {
// Set up llama's logging
llama_log_set(my_log_callback, NULL);

// Redirect mtmd/clip logging to use the same callback
mtmd_log_set_llama_callback(my_log_callback, NULL);

// Now all mtmd and clip logs will use your custom callback
mtmd_context * ctx = mtmd_init_from_file(...);
// ...
}
```

This ensures that all logging from `libmtmd`, including the underlying `clip.cpp` vision encoder, is routed through your application's logging system consistently.

## How to obtain `mmproj`

Multimodal projector (`mmproj`) files are specific to each model architecture.
Expand Down
7 changes: 7 additions & 0 deletions tools/mtmd/clip-impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,13 @@ struct clip_logger_state {

extern struct clip_logger_state g_logger_state;

// Function to set logging callback (can be used to redirect to llama's logging)
// If not called, will use the default callback (logs to stderr)
static inline void clip_log_set_callback(ggml_log_callback callback, void * user_data) {
g_logger_state.log_callback = callback;
g_logger_state.log_callback_user_data = user_data;
}

static void clip_log_internal_v(enum ggml_log_level level, const char * format, va_list args) {
if (format == NULL) {
return;
Expand Down
6 changes: 5 additions & 1 deletion tools/mtmd/clip.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,11 @@
#include <numeric>
#include <functional>

struct clip_logger_state g_logger_state = {GGML_LOG_LEVEL_CONT, clip_log_callback_default, NULL};
struct clip_logger_state g_logger_state = {
GGML_LOG_LEVEL_CONT, // verbosity_thold
clip_log_callback_default, // log_callback
NULL // log_callback_user_data
};

enum ffn_op_type {
FFN_GELU,
Expand Down
4 changes: 4 additions & 0 deletions tools/mtmd/mtmd.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,10 @@ void mtmd_free(mtmd_context * ctx) {
}
}

void mtmd_log_set_llama_callback(ggml_log_callback llama_cb, void * llama_user_data) {
clip_log_set_callback(llama_cb, llama_user_data);
}

struct mtmd_tokenizer {
mtmd_context * ctx;
std::vector<const mtmd_bitmap *> bitmaps;
Expand Down
8 changes: 8 additions & 0 deletions tools/mtmd/mtmd.h
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,14 @@ MTMD_API mtmd_context * mtmd_init_from_file(const char * mmproj_fname,

MTMD_API void mtmd_free(mtmd_context * ctx);

// Set up logging to use llama's logging callback
// This redirects all mtmd/clip logging through llama's logging system
// Call this after llama_log_set to ensure mtmd uses the same logging callback
// Example:
// llama_log_set(my_log_callback, my_user_data);
// mtmd_log_set_llama_callback(my_log_callback, my_user_data);
MTMD_API void mtmd_log_set_llama_callback(ggml_log_callback llama_cb, void * llama_user_data);

// whether we need to set non-causal mask before llama_decode
MTMD_API bool mtmd_decode_use_non_causal(mtmd_context * ctx);

Expand Down
Loading