Extracting hidden_states

[orannahum-qualifire]

I want to explore the values inside GGUF models and wanted to know if there’s a way to extract the hidden_states?

Like in transformers where you do:

# Example input
inputs = tokenizer(example_input, return_tensors="pt")

# Move to model device if needed
inputs = {k: v.to(model.device) for k, v in inputs.items()}

# Forward pass through the base model only
with torch.no_grad():
    # This returns the hidden states from the last layer
    hidden_states = model.model(**inputs)  

Is it possible to do the same with llama.cpp?

[ryan-mangeno]

I think you might be looking for the

llama-eval-callback 

tool, you can run it with a -m flag to pass in the path to your gguf model