Skip to content

Conversation

@mollyxu
Copy link
Contributor

@mollyxu mollyxu commented Nov 20, 2025

Remove skipped tests from collection in internal environment

Prevent internal CI failures from skipped tests by excluding skipped tests from collection in internal environment.

TorchCodec tests were flaky / broken internal because:

  • Tests were using runtime pytest.skip() calls, so they were collected but then skipped during execution

Changes Made

1. Added @needs_ffmpeg_cli marker to tests that need ffmpeg-cli

  • test_ops.py::TestVideoDecoderOps::test_seek_mode_custom_frame_mappings
  • test_ops.py::TestVideoDecoderOps:: test_seek_mode_custom_frame_mappings_fails
  • test_encoders.py::TestAudioEncoder::test_against_cli
  • test_encoders.py::TestAudioEncoder::test_video_encoder_against_ffmpeg_cli
  • test_encoders.py::TestAudioEncoder::test_codec_parameter_utilized
  • test_encoders.py::TestAudioEncoder::test_codec_spec_vs_impl_equivalence
  • test_encoders.py::TestAudioEncoder::test_extra_options_utilized
  • test_decoders.py::TestVideoDecoder::test_custom_frame_mappings_json_and_bytes
  • test_decoders.py::TestVideoDecoder::test_custom_frame_mappings_init_fails

2. Converted runtime skips to collection-time skips

Replaced runtime pytest.skip() calls with pytest.param(..., marks=pytest.mark.skipif(...)) for:

  • FFmpeg version compatibility checks (e.g., FFmpeg 4 incompatibility with wav files)
  • Platform-specific codec availability (e.g., webm codec unavailable on Windows with certain FFmpeg versions)

3. Updated conftest.py to exclude tests internally

Modified pytest_collection_modifyitems hook to exclude tests from collection when running internally if they have:

  • @needs_ffmpeg_cli marker
  • @pytest.mark.skip() decorator
  • @pytest.mark.skipif() decorator

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 20, 2025
@meta-codesync
Copy link

meta-codesync bot commented Nov 21, 2025

@mollyxu has imported this pull request. If you are a Meta employee, you can view this in D87646776.

@mollyxu mollyxu marked this pull request as ready for review November 21, 2025 17:03
if get_ffmpeg_major_version() == 4 and format == "wav":
pytest.skip("Swresample with FFmpeg 4 doesn't work on wav files")
if IS_WINDOWS and get_ffmpeg_major_version() <= 5 and format == "mp3":
# TODO: https://github.com/pytorch/torchcodec/issues/837
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to preserve the comment pointing to an issue?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think line 371 is a good place to move this TODO?

Copy link
Contributor

@scotts scotts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thank you for fixing this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants