Support model deployment

[arcticfly]

For models that Together supports as serverless endpoints, we can now deploy LoRAs to Together and query them.

Changes

• Document HUGGINGFACE_TOKEN and HUGGING_FACE_HUB_TOKEN, used for training gated models
• Document YOUR_TOGETHER_API_KEY, used for uploading models to Together
• Update all tic tac toe rollouts to use scenario classes
• Only report chat tic tac toe completions after trajectory completes
• Create utility function get_step_checkpoint_dir
• Add archive_and_presign_step_url for preparing checkpoint dir to upload to Together
• Add _experimental_deploy function to LocalBackend and Backend
• Add utils to manage Together deployments
• Update tic-tac-toe-local.py to deploy a model to Together and use it in a rollout
• Add UnsupportedBaseModelDeploymentError and LoRADeploymentTimedOutError errors
• Rewrite Together job statuses into simpler LoRADeploymentJobStatusBody type

Some relevant types:

class LoRADeploymentProvider(str, Enum):
    TOGETHER = "together"


class LoRADeploymentJobStatus(str, Enum):
    QUEUED = "Queued"
    RUNNING = "Running"
    COMPLETE = "Complete"
    FAILED = "Failed"


class LoRADeploymentJobStatusBody(BaseModel):
    status: LoRADeploymentJobStatus
    job_id: str
    model_name: str
    failure_reason: str | None

TODO:
Test _experimental_deploy through SkyPilot

[corbt]

I think those token names have been deprecated and replaced with HF_TOKEN. https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables

[corbt]

Why the big uv.lock diff? Doesn't look like pyproject.toml changed?

[arcticfly]

Why the big uv.lock diff? Doesn't look like pyproject.toml changed?

I assume someone ran uv sync on mac

[corbt]

Let's add a required parameter deploy_to, which must be set to "together". Will help people understand what's going on.

[arcticfly]

Added this class:

class LoRADeploymentProvider(str, Enum):
    TOGETHER = "together"

[corbt]

Can we add an explicit check in this function that the base model is one that Together supports? I guess the user will figure it out eventually if they try one that isn't supported but would be nice to have it explicitly up front so you can take that into account when deciding which base model to train.

[arcticfly]

Now returning a UnsupportedBaseModelDeploymentError error.

[corbt]

I assume someone ran uv sync on mac

I'm not an expert but according to this uv.lock is supposed to be cross-platform: https://docs.astral.sh/uv/guides/projects/#pyprojecttoml

[arcticfly]

I assume someone ran uv sync on mac

I'm not an expert but according to this uv.lock is supposed to be cross-platform: https://docs.astral.sh/uv/guides/projects/#pyprojecttoml

Might not be a platform issue, but I have noticed that running on my local mac messes up the uv lock and interferes with installation on a linux machine. Worth looking into, probably not in scope for this PR.

[corbt]

nice work!