Example of using Claude 3 sonnet V2 on AWS Bedrock

[TroyZuroske]

Can anyone provide a script of how to get Claude 3 Sonnet V2 on AWS Bedrock to work with the library? I think I am missing a parameter or something. When I run this script with my AWS environment variables set, it starts working and is able to find my credentials but when it gets to generating topics, it crashes because "The maximum tokens you requested exceeds the model limit of 8192. Try again with a maximum tokens value that is lower than 8192."

Simple script:

from annotateai import Annotate

annotate = Annotate("bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0")  

annotate("https://arxiv.org/pdf/2406.14657")

Summarized Traceback:

10:51:27 - LiteLLM:INFO: utils.py:952 - Wrapper: Completed Call, calling success_handler
2025-01-21 10:51:27,736 - LiteLLM - INFO - Wrapper: Completed Call, calling success_handler
Generating keywords: 100%|██████████| 1/1 [00:01<00:00,  1.72s/it]
Generating annotations: 100%|██████████| 24/24 [00:17<00:00,  1.41it/s]
Generating topics:   0%|          | 0/106 [00:00<?, ?it/s]10:51:44 - LiteLLM:INFO: utils.py:2820 - 
LiteLLM completion() model= anthropic.claude-3-5-sonnet-20241022-v2:0; provider = bedrock
2025-01-21 10:51:44,816 - LiteLLM - INFO - 
LiteLLM completion() model= anthropic.claude-3-5-sonnet-20241022-v2:0; provider = bedrock
2025-01-21 10:51:45,118 - httpx - INFO - HTTP Request: POST https://bedrock-runtime.us-west-2.amazonaws.com/model/anthropic.claude-3-5-sonnet-20241022-v2:0/converse "HTTP/1.1 400 Bad Request"
raise BadRequestError(
litellm.exceptions.BadRequestError: litellm.BadRequestError: BedrockException - {"message":"The maximum tokens you requested exceeds the model limit of 8192. Try again with a maximum tokens value that is lower than 8192."}

[davidmezzetti]

It looks like this is the issue: https://github.com/neuml/annotateai/blob/master/src/python/annotateai/annotate.py#L237

It looks like the maxlength needs to be configurable. Alternatively, there could also be a change put into txtai to limit the max length based on the model max length (https://docs.litellm.ai/docs/completion/token_usage#7-get_max_tokens).