Spring AI vertext AI Gemini response interleaving text and images

[adn911]

Hello,

We are using Spring AI with Vertex AI API to call Gemini. Now for some responses, we expect both text as well as Images (base64). I have tried using ChatClient to retrieve the whole response containing interleaving text and images(base64), but couldn't do it so far, the image does not render. Can anyone guide me how to do this using Spring AI?

Thank you in advance.

[UsmanovMahmudkhan]

Hi,

For this use case, ChatClient may not be the right abstraction because chatClient.call().content() is mainly designed to return text content. If Gemini returns mixed output such as text plus generated images, the image is usually returned as a separate response part, not as normal text.

With Vertex AI Gemini, image output normally comes through inlineData parts. So the flow should be:

1. Use a Gemini model that supports image generation.
2. Configure the request with both text and image response modalities:

"responseModalities": ["TEXT", "IMAGE"]

3. Do not rely only on .content().
4. Read the full/raw response parts from the candidate.
5. Extract text parts separately from inlineData image parts.
6. Convert the image bytes to Base64 or a data URL, then render it in the frontend.

Example structure:

for (Part part : response.getCandidates(0).getContent().getPartsList()) {
    if (part.hasText()) {
        String text = part.getText();
    }

    if (part.hasInlineData()) {
        String mimeType = part.getInlineData().getMimeType();

        String base64 = Base64.getEncoder().encodeToString(
            part.getInlineData().getData().toByteArray()
        );

        String imageUrl = "data:" + mimeType + ";base64," + base64;
    }
}

Then the frontend can render it like this:

<img src="data:image/png;base64,BASE64_IMAGE_HERE" />

or dynamically:

img.src = `data:${mimeType};base64,${base64}`;

So the issue is probably not that the image “does not render” directly. The real issue is that the Spring AI ChatClient text abstraction may not expose or preserve the non-text response parts in the way needed.

If Spring AI’s VertexAiGeminiChatModel does not currently expose the raw inlineData parts from Gemini responses, the practical solution is to use the lower-level Vertex AI / Google GenAI client for this specific image-generation flow, and keep Spring AI ChatClient for normal text-only chat flows.

[Jeevannaik66]

Have you looked at what the response payload is actually returning? If the Gemini API is sending back image data correctly, the problem might lie in how Spring AI handles the multimodal response, not in Vertex AI itself. I recommend checking the raw response first to see if the image parts are present as base64 data. If they are, you might need to process the individual content parts instead of depending on the default text response handling from ChatClient. Also, which version of Spring AI are you using? It would be helpful to see a small code example of how you're calling Gemini and how you're trying to display the image content.

[MILLERMARRU]

The issue is that .content() only extracts text parts — for multimodal responses containing images you need to go through .chatResponse() and access the media list on the AssistantMessage.

Working code

import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.vertexai.gemini.VertexAiGeminiChatOptions;
import org.springframework.util.MimeType;

import java.util.Base64;

@Service
public class GeminiMultimodalService {

    private final ChatClient chatClient;

    public GeminiMultimodalService(ChatClient.Builder builder) {
        this.chatClient = builder
            .defaultOptions(VertexAiGeminiChatOptions.builder()
                .model("gemini-2.0-flash-001")
                // Tell Gemini to return both text and images
                .responseModalities(List.of("TEXT", "IMAGE"))
                .build())
            .build();
    }

    public record MultimodalResult(String text, List<String> imageDataUrls) {}

    public MultimodalResult generate(String prompt) {
        ChatResponse response = chatClient.prompt()
            .user(prompt)
            .call()
            .chatResponse();

        List<String> images = new ArrayList<>();
        StringBuilder text = new StringBuilder();

        for (var generation : response.getResults()) {
            AssistantMessage msg = generation.getOutput();

            // Text part
            if (msg.getText() != null) {
                text.append(msg.getText());
            }

            // Image parts — each media item has a MimeType and byte[] data
            for (AssistantMessage.Media media : msg.getMedia()) {
                MimeType mimeType = media.getMimeType();
                if (mimeType.getType().equals("image")) {
                    byte[] imageBytes = (byte[]) media.getData();
                    String base64 = Base64.getEncoder().encodeToString(imageBytes);
                    // Ready to embed in an <img src="..."> tag
                    images.add("data:" + mimeType + ";base64," + base64);
                }
            }
        }

        return new MultimodalResult(text.toString(), images);
    }
}

REST endpoint example

@RestController
@RequestMapping("/api/generate")
public class GenerateController {

    @Autowired
    private GeminiMultimodalService service;

    @PostMapping
    public ResponseEntity<Map<String, Object>> generate(@RequestBody Map<String, String> body) {
        var result = service.generate(body.get("prompt"));
        return ResponseEntity.ok(Map.of(
            "text", result.text(),
            "images", result.imageDataUrls()  // send data URLs to frontend
        ));
    }
}

Frontend rendering (React / plain HTML)

// React
{result.images.map((dataUrl, i) => (
    <img key={i} src={dataUrl} alt={`Generated image ${i + 1}`} />
))}

Key points

• responseModalities must explicitly include "IMAGE" — Gemini defaults to text-only output.
• media.getData() returns byte[] for inline image data. Cast it explicitly before encoding.
• If you're using Gemini 2.0 Flash and the image comes back empty, verify your Vertex AI project has the imagegeneration API enabled and your model version supports native image output (not all Gemini models do).
• For large images, consider streaming the bytes to a GCS bucket and returning a signed URL rather than embedding base64 inline.