Design question: annotation-based LLM cost guardrails in Spring Boot

[AbaSheger]

Hi everyone,

I’ve been experimenting with a small open-source Spring Boot starter called Guardrail4J.

The idea is to add cost guardrails around LLM calls in Java/Spring apps using an annotation-based approach:

@LLMGuarded(
    provider = "openai",
    model = "gpt-4o-mini",
    userId = "#userId",
    tenantId = "#tenantId",
    feature = "document-summary"
)

The starter currently uses Spring AOP to intercept the method, estimate cost, apply budget rules, and then ALLOW, WARN, BLOCK, or FALLBACK.
I’m trying to understand the best direction for this kind of tool in the Spring ecosystem.
Would you expect LLM cost control to be handled through:
1.annotations like this
2. configuration/properties
3.Spring AI integration
4 .gateway/proxy layer
5. something else?
The project is still an early MVP, not production-ready. I’m mainly looking for design feedback from Java/Spring developers.
Repo: https://github.com/AbaSheger/guardrail4j⁠

[Jeevannaik66]

Hi everyone,

I’ve been experimenting with a small open-source Spring Boot starter called Guardrail4J.

The idea is to add cost guardrails around LLM calls in Java/Spring apps using an annotation-based approach:

@LLMGuarded(
    provider = "openai",
    model = "gpt-4o-mini",
    userId = "#userId",
    tenantId = "#tenantId",
    feature = "document-summary"
)

The starter currently uses Spring AOP to intercept the method, estimate cost, apply budget rules, and then ALLOW, WARN, BLOCK, or FALLBACK.
I’m trying to understand the best direction for this kind of tool in the Spring ecosystem.
Would you expect LLM cost control to be handled through:
1.annotations like this
2. configuration/properties
3.Spring AI integration
4 .gateway/proxy layer
5. something else?
The project is still an early MVP, not production-ready. I’m mainly looking for design feedback from Java/Spring developers.
Repo: https://github.com/AbaSheger/guardrail4j⁠

This project is interesting. I like the annotation-based approach because it keeps cost guidelines close to the code. This makes it easy for developers to see which LLM operations are protected. However, I would avoid placing budget limits or model details directly in annotations since those values often change. Configuration files seem like a better place for that information.
I also think Spring AI integration would be helpful because many teams building AI features in Spring Boot are already using it. This could simplify adoption and cut down on custom setup. Overall, I see annotations as the layer that developers interact with. Meanwhile, configuration and Spring AI integration manage the actual policies. This setup strikes a good balance between simplicity, flexibility, and maintainability.

[AbaSheger]

Thanks, this is exactly the kind of feedback I was looking for.

I agree with your point about not putting changing values like model names, provider names, and budget limits directly in the annotation. The annotation should probably stay close to the code to mark which LLM operation is protected, while the actual policy should live in configuration.

Something like:

@LLMGuarded(policy = "document-summary", userId = "#userId", tenantId = "#tenantId")

and then the model, provider, limits, and actions are defined in application.yml.

I also agree that Spring AI integration is probably the next important direction, especially for teams already using ChatClient and advisors.

So the direction I’m leaning toward now is:

1. annotations for developer-facing intent
2. configuration for mutable policies
3. Spring AI Advisor integration for native Spring AI usage

Really appreciate the thoughtful feedback.