Skip to content

Conversation

robotrapta
Copy link
Member

No description provided.

Copy link
Member

@sunildkumar sunildkumar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! left some questions.

# Check if the stop string is in the completions
# We need to convert the tensor to a string.
if self.tool_defn.completion_has_tool_call(prompt_completion_str):
tool_response_str = self.tool_defn.call_tool(prompt_completion_str)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why doesn't the dataclass have call_tool?

# We need to convert the tensor to a string.
if self.tool_defn.completion_has_tool_call(prompt_completion_str):
tool_response_str = self.tool_defn.call_tool(prompt_completion_str)
tool_response_ids_list = self.processing_class.tokenizer.encode(tool_response_str, add_special_tokens=False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming this doesn't add an extra BOS token?

return inputs

def _generate_completion(
self, model: PreTrainedModel, prompt_inputs: dict[str, torch.Tensor]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, thanks for making this a function, clearly the right move.

return prompt_completion_ids

def _generate_single_completion_with_tools(
self, model: PreTrainedModel, prompt_inputs: dict[str, torch.Tensor], max_steps: int = 10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same nit here - BatchFeature

(Note that 46*44 is 2024).
"""
conv = SingleConversationWithTools(prompt_inputs, self.tool_defn, self.processing_class)
# Loop until tool isn't called, of we max out
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Loop until tool isn't called, of we max out
# Loop until tool isn't called, or we max out

- input_ids: [1, 710] ints. Some stuff at the beginning and the end, the middle full of 151655
- attention_mask: [1, 710] ints. All 1
- pixel_values: 2024x1176 floats. The image.
- image_grid_thw: a 1x3 tensor with values: [1, 46, 44].
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe add a short comment about what max _steps is.

My understanding: The generation will stop once a tool is called, then this code processes the tool call. max_steps is the maximum number of tools we're willing to process for a single completion?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants