Skip to content

SageMakerBaseOperator character limit bug #45550

Description

@dirkrkotzeml

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

8.27.0

Apache Airflow version

2.7.2

Operating System

Amazon Linux

Deployment

Amazon (AWS) MWAA

Deployment details

Standard MWAA deployment - V2.7.2

What happened

SageMakerProcessingJobs have a hard limit of 64 characters for the ProcessingJobName.
In the SageMakerBaseOperator there is a check for uniqueness for the name.
In the case that a name is not unique it adds a timestamp to prevent a potential collision, however there is no check to prevent the updated - from exceeding 64 characters. This causes the creation of the SageMakerProcessingJob to fail.

What you think should happen instead

In the SageMaker Pipelines SDK they truncate the base name before adding the timestamp, therefor we recommend taking a similar approach for consistency purposes.

How to reproduce

Create a SageMaker ProcessingJob using the SageMakerProcessingOperator with a name of longer than 50 characters, and trigger it more than once. On the second time it is triggered the time stamp will be added and in the airflow logs it will show the error stating the sagemaker processing job failed to create due to the name exceeding the character limit

Anything else

Every time that a scheduled run occurs with the same name (after the first run)

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions