-
Notifications
You must be signed in to change notification settings - Fork 2
Open
Description
Description
Currently, logs only output to stdout. We should add the ability to log to a file
and automatically save that log file to MLflow for better debugging and analysis.
Context
When runs encounter errors or need debugging, having a complete log file saved in
MLflow would be valuable for post-mortem analysis and tracking training history.
Requirements
-
Configure Python's logging library to write to both stdout and a file
- Research the logging library's configuration for multiple handlers
- Add a file handler in addition to the existing stream handler
-
Add log file path configuration
- Option 1: Add a
log_file_pathparameter to the run configuration - Option 2: Use a default path convention (e.g.,
logs/{experiment_name}/{run_name}.log)
- Option 1: Add a
-
Integrate with simplexity run manager
- Ensure log file is saved to MLflow in the cleanup/teardown phase
- Include log file upload even when runs fail/error out
- Log file should be available as an MLflow artifact
Acceptance Criteria
- Logs are written to both stdout and a file simultaneously
- Log file path is configurable (either via config or using sensible defaults)
- Log files are automatically uploaded to MLflow as artifacts
- Log files are captured and uploaded even when runs fail with errors
- Documentation updated to explain log file configuration options
Notes
The log file path needs to be known by the run manager's cleanup function to ensure
upload happens reliably.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels