feat(trainer): add structured and configurable logging support #110

anencore94 · 2025-09-27T15:21:05Z

What this PR does / why we need it:

This PR implements comprehensive structured and configurable logging support for the Kubeflow SDK as requested in Issue #85. The implementation provides:

NullHandler Pattern: Prevents logging noise by default while allowing users to override with their own configuration
Configurable Logging: Support for console, detailed, and JSON output formats
Structured Logging: JSON formatter for log aggregation systems (ELK stack, Fluentd, etc.)
Environment-based Configuration: Configure logging via environment variables
SDK Integration: Added debug logging to TrainerClient for better observability

The logging system ensures that the SDK is quiet by default (using NullHandler) but provides rich logging capabilities when users explicitly configure logging.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #85

Features Implemented

Core Logging Infrastructure

NullHandler Pattern: Prevents logging noise by default, users can override with their own configuration
Configurable Logging: Support for console, detailed, and JSON output formats
Structured Logging: JSON formatter for log aggregation systems (ELK stack, Fluentd, etc.)
Environment-based Configuration: Configure logging via environment variables

Key Components

kubeflow/trainer/logging/config.py: Centralized logging configuration
kubeflow/trainer/logging/formatters.py: Custom formatters including JSON structured logging
kubeflow/trainer/logging/logging_test.py: Comprehensive test suite (16 tests)

Integration Points

SDK Integration: Added debug logging to TrainerClient for better observability
Package Integration: Exposed logging utilities through kubeflow.trainer.logging
NullHandler Setup: Configured at package level to prevent logging noise

Issue #85 Requirements Fulfilled

✅ Consistent use of Python's logging library instead of print statements

All SDK operations now use proper Python logging
NullHandler pattern prevents unwanted output by default

✅ Support for different logging levels (DEBUG, INFO, WARNING, ERROR)

Full support for all standard Python logging levels
Configurable via setup_logging() function or environment variables

✅ Ability for users to configure log formatting and destinations

Console, detailed, and JSON output formats
File output support
Custom formatter support

✅ Clear and actionable log messages for key SDK operations

Debug messages in TrainerClient initialization
Backend selection logging
Job creation and ID logging

Testing

16 comprehensive unit tests covering all logging functionality
NullHandler pattern verification with proper isolation
SDK integration testing with real TrainerClient usage
Application integration examples with file and console logging
All tests pass with proper linting compliance

Usage Examples

Basic Usage

from kubeflow.trainer import TrainerClient, setup_logging

# Setup logging (optional - NullHandler prevents noise by default)
setup_logging(level="DEBUG", format_type="console")

# Use SDK - debug messages will appear if logging is configured
client = TrainerClient()

JSON Logging for Production

setup_logging(level="INFO", format_type="json")
# Logs will be in JSON format suitable for log aggregation

Environment Configuration

export KUBEFLOW_LOG_LEVEL=DEBUG
export KUBEFLOW_LOG_FORMAT=json

Breaking Changes

None - this is a pure addition with backward compatibility.

Migration Guide

No migration required. Existing code will continue to work without changes.
Users can optionally configure logging for better observability.

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2025-09-27T15:21:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This PR implements comprehensive logging support for the Kubeflow SDK as requested in Issue kubeflow#85. ## Features Implemented ### Core Logging Infrastructure - **NullHandler Pattern**: Prevents logging noise by default, users can override with their own configuration - **Configurable Logging**: Support for console, detailed, and JSON output formats - **Structured Logging**: JSON formatter for log aggregation systems (ELK stack, Fluentd, etc.) - **Environment-based Configuration**: Configure logging via environment variables ### Key Components - `kubeflow/trainer/logging/config.py`: Centralized logging configuration - `kubeflow/trainer/logging/formatters.py`: Custom formatters including JSON structured logging - `kubeflow/trainer/logging/logging_test.py`: Comprehensive test suite (14 tests) ### Integration Points - **SDK Integration**: Added debug logging to TrainerClient for better observability - **Package Integration**: Exposed logging utilities through kubeflow.trainer.logging - **NullHandler Setup**: Configured at package level to prevent logging noise ## Issue kubeflow#85 Requirements Fulfilled ✅ **Consistent use of Python's logging library instead of print statements** - All SDK operations now use proper Python logging - NullHandler pattern prevents unwanted output by default ✅ **Support for different logging levels (DEBUG, INFO, WARNING, ERROR)** - Full support for all standard Python logging levels - Configurable via setup_logging() function or environment variables ✅ **Ability for users to configure log formatting and destinations** - Console, detailed, and JSON output formats - File output support - Custom formatter support ✅ **Clear and actionable log messages for key SDK operations** - Debug messages in TrainerClient initialization - Backend selection logging - Job creation and ID logging ## Testing - **14 comprehensive unit tests** covering all logging functionality - **NullHandler pattern verification** with proper isolation - **SDK integration testing** with real TrainerClient usage - **Application integration examples** with file and console logging - **All tests pass** with proper linting compliance ## Usage Examples ### Basic Usage ```python from kubeflow.trainer import TrainerClient, setup_logging # Setup logging (optional - NullHandler prevents noise by default) setup_logging(level="DEBUG", format_type="console") # Use SDK - debug messages will appear if logging is configured client = TrainerClient() ``` ### JSON Logging for Production ```python setup_logging(level="INFO", format_type="json") # Logs will be in JSON format suitable for log aggregation ``` ### Environment Configuration ```bash export KUBEFLOW_LOG_LEVEL=DEBUG export KUBEFLOW_LOG_FORMAT=json ``` ## Breaking Changes None - this is a pure addition with backward compatibility. ## Migration Guide No migration required. Existing code will continue to work without changes. Users can optionally configure logging for better observability. Resolves kubeflow#85 Signed-off-by: Jaeyeon Kim <[email protected]>

coveralls · 2025-09-27T16:43:36Z

Pull Request Test Coverage Report for Build 18062345345

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 71.739%

Totals
Change from base Build 17979146135:	0.0%
Covered Lines:	297
Relevant Lines:	414

💛 - Coveralls

andreyvelich

Thank you for this @anencore94!
/assign @kubeflow/kubeflow-sdk-team

andreyvelich · 2025-10-16T19:18:09Z

kubeflow/trainer/__init__.py

+    "get_logger",
+    "setup_logging",


shall we move this to a separate package (e.g. kubeflow/logging)

andreyvelich · 2025-10-16T19:20:34Z

kubeflow/trainer/api/trainer_client.py

+        )
+
+        job_id = self.backend.train(runtime=runtime, initializer=initializer, trainer=trainer)
+        logger.debug("Successfully created TrainJob with ID: %s", job_id)


Can you move all of the debug statements out of backend, and place them into trainer_client.py

sdk/kubeflow/trainer/backends/kubernetes/backend.py

Lines 254 to 256 in 9642b41

logger.debug(

f"{constants.TRAINJOB_KIND} {self.namespace}/{train_job_name} has been created"

)

@szaher @kramaranya Does it sound good ?

andreyvelich · 2025-10-16T19:21:22Z

kubeflow/trainer/logging/config.py

+
+
+def setup_logging(
+    level: Union[str, int] = "INFO",


As we discussed here, do we need to have INFO level for our logger ? #85 (comment)

andreyvelich · 2025-10-16T19:24:54Z

kubeflow/trainer/logging/config.py

+        "formatters": {
+            "console": {
+                "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+                "datefmt": "%Y-%m-%d %H:%M:%S",


Maybe it is better to make fmt ISO compliant ?

Suggested change

"datefmt": "%Y-%m-%d %H:%M:%S",

"datefmt": "%Y-%m-%dT%H:%M:%S",

andreyvelich · 2025-10-16T19:25:05Z

kubeflow/trainer/logging/config.py

+                "format": (
+                    "%(asctime)s - %(name)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s"
+                ),
+                "datefmt": "%Y-%m-%d %H:%M:%S",


Suggested change

"datefmt": "%Y-%m-%d %H:%M:%S",

"datefmt": "%Y-%m-%dT%H:%M:%S",

andreyvelich · 2025-10-16T19:26:35Z

kubeflow/trainer/logging/logging_test.py

+class TestLoggingConfig:
+    """Test logging configuration functionality."""
+
+    def test_get_logger(self):


Please refactor these test cases, similar to this:

sdk/kubeflow/trainer/backends/kubernetes/backend_test.py

Line 573 in 9642b41

@pytest.mark.parametrize(

andreyvelich · 2025-10-16T19:28:13Z

kubeflow/trainer/api/trainer_client.py

            ValueError: Invalid backend configuration.

        """
+        logger.debug("Initializing TrainerClient with backend_config=%s", backend_config)


How you propagate your loggers to the clients ? Should it be part of the TrainerClient() property ?

google-oss-prow bot requested review from kramaranya and szaher September 27, 2025 15:21

google-oss-prow bot added the size/XL label Sep 27, 2025

anencore94 force-pushed the feat/logging-system-v2 branch from 149c8bc to 56896c3 Compare September 27, 2025 15:23

anencore94 changed the title ~~feat(logging): Implement structured and configurable logging support~~ feat: add structured and configurable logging support Sep 27, 2025

anencore94 marked this pull request as draft September 27, 2025 15:24

google-oss-prow bot added the do-not-merge/work-in-progress label Sep 27, 2025

anencore94 changed the title ~~feat: add structured and configurable logging support~~ feat(trainer): add structured and configurable logging support Sep 27, 2025

anencore94 marked this pull request as ready for review September 27, 2025 15:55

google-oss-prow bot removed the do-not-merge/work-in-progress label Sep 27, 2025

anencore94 force-pushed the feat/logging-system-v2 branch 2 times, most recently from 575c624 to 6f99044 Compare September 27, 2025 16:35

anencore94 force-pushed the feat/logging-system-v2 branch from 6f99044 to 9642b41 Compare September 27, 2025 16:41

andreyvelich reviewed Oct 16, 2025

View reviewed changes

andreyvelich mentioned this pull request Oct 20, 2025

feat: Add ContainerBackend with Docker and Podman #119

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(trainer): add structured and configurable logging support #110

feat(trainer): add structured and configurable logging support #110

anencore94 commented Sep 27, 2025 •

edited

Loading

Uh oh!

google-oss-prow bot commented Sep 27, 2025

Uh oh!

coveralls commented Sep 27, 2025

Uh oh!

andreyvelich left a comment

Uh oh!

andreyvelich Oct 16, 2025

Uh oh!

andreyvelich Oct 16, 2025

Uh oh!

andreyvelich Oct 16, 2025

Uh oh!

andreyvelich Oct 16, 2025

Uh oh!

andreyvelich Oct 16, 2025

Uh oh!

andreyvelich Oct 16, 2025

Uh oh!

andreyvelich Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	logger.debug(
	f"{constants.TRAINJOB_KIND} {self.namespace}/{train_job_name} has been created"
	)

	"datefmt": "%Y-%m-%d %H:%M:%S",
	"datefmt": "%Y-%m-%dT%H:%M:%S",

feat(trainer): add structured and configurable logging support #110

Are you sure you want to change the base?

feat(trainer): add structured and configurable logging support #110

Conversation

anencore94 commented Sep 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Features Implemented

Core Logging Infrastructure

Key Components

Integration Points

Issue #85 Requirements Fulfilled

Testing

Usage Examples

Basic Usage

JSON Logging for Production

Environment Configuration

Breaking Changes

Migration Guide

Uh oh!

google-oss-prow bot commented Sep 27, 2025

Uh oh!

coveralls commented Sep 27, 2025

Pull Request Test Coverage Report for Build 18062345345

Details

💛 - Coveralls

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anencore94 commented Sep 27, 2025 •

edited

Loading