Remove telemetry, SmartDashboard, and indirect entrypoint functionality #789
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Remove TelemetryConfiguration classes and related code - Remove telemetry monitor entrypoint and utilities - Remove telemetry collectors and sinks - Remove telemetry-related tests - Remove watchdog dependency - Simplify job entities and controller logic - Remove telemetry configuration from config.py This removes approximately 5,838 lines of telemetry-related code while preserving core SmartSim functionality.
- Remove telemetry_dir usage from controller.py batch job creation - Clean up telemetry references in job.py comments and docstrings - Remove telemetry-related properties from manifest.py - Update serialize.py to remove telemetry directory and metadata references - Remove telemetry_dir argument from indirect.py entrypoint and step.py launcher - Update indirect tests to remove telemetry_dir parameter expectations - Fix conftest.py to import JobEntity from correct location - Clean up remaining telemetry comments and replace with generic logging All telemetry code, configuration, tests, and documentation have now been completely removed from the SmartSim codebase.
- Clean up remaining telemetry references in job.py comments - Simplify step.py proxy decorator to always use direct launch - Remove telemetry.disable() call from CLI validate.py - Simplify dragon backend cooldown period configuration - Remove unused get_config import from dragon backend All telemetry code has been completely removed from SmartSim. The codebase now works without any telemetry dependencies or references.
- Replace CONFIG.telemetry_subdir references with 'status' directory - Remove telemetry event tracking from test_process_failure and test_complete_process - Simplify tests to focus on actual process execution rather than telemetry events - All indirect tests now pass without telemetry dependencies Tests now verify core functionality without relying on removed telemetry system.
- Remove dashboard CLI plugin and all associated functionality - Remove SmartDashboard documentation file (smartdashboard.rst) - Update documentation index to remove SmartDashboard section - Clean up ReadTheDocs configuration to remove dashboard dependency - Update Docker files to remove SmartDashboard installation - Remove dashboard-related tests and update plugin tests - Update changelog to document SmartDashboard removal as breaking change - Remove SmartDashboard changelog section SmartSim now operates independently without SmartDashboard integration. The core monitoring and logging functionality is preserved through SmartSim's existing logging infrastructure.
- Add proper type annotation for empty plugins tuple in plugin.py - Add explicit type annotation for plugin_items in cli.py - All mypy checks now pass successfully
- Remove telemetry-related test functions from test_experiment.py - Fix status_dir metadata by setting it to .smartsim subdirectory - Fix controller test expecting removed exp_path parameter - All tests now pass and mypy is clean
- Remove telemetry-related test functions from test_config.py and test_serialize.py - Remove telemetry fixtures and references from test_logs.py and conftest.py - Update manifest_json fixture to use simple path instead of telemetry_subdir - All tests now pass without telemetry dependencies
- Updated test_output_files.py to match simplified .smartsim directory structure - Updated test_symlinking.py to use new output file paths - Fixed controller to use absolute paths for status directories - Implemented historical file preservation with timestamps - Updated batch job tests to use correct entity relationships - Modified symlink_error test to match new auto-creating behavior All core telemetry removal is complete with only output redirection issues remaining.
- Remove unused imports (CONFIG, subprocess, sys, pathlib, get_ts_ms, encode_cmd, UnproxyableStepError) - Fix line length issues in indirect.py and job.py - Remove unreachable code after return statements - Remove unused variables (start_rc, status_dir, is_dragon) - Fix import-outside-toplevel issue with time module in controller.py - Add pylint disable comment for unused argument raw_experiment - Remove unnecessary pass statement and simplify docstring All lint checks now pass with 10.00/10 rating.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #789 +/- ##
===========================================
- Coverage 83.91% 80.27% -3.64%
===========================================
Files 83 78 -5
Lines 6284 6090 -194
===========================================
- Hits 5273 4889 -384
- Misses 1011 1201 +190
🚀 New features to boost your workflow:
|
- Delete smartsim/_core/entrypoints/indirect.py - Delete tests/test_indirect.py - Update step.py comment to remove references to indirect launching - Clean up cached files and mypy cache for removed modules - Verified all tests pass and no type errors remain
- Fix KeyError for status directory in batch job steps by setting status_dir in _create_batch_job_step - Remove test_orc_telemetry test that referenced deleted telemetry functionality - Remove remaining telemetry environment variable settings from dragon and pals tests - Update line formatting for better lint compliance - All originally failing tests now pass
al-rigazzi
changed the title
- Enhanced symlink_output_files to auto-create parent directories - Fixed path handling for entities with sub-entities (Orchestrator/Ensemble) - Ensured all tests use proper test directories instead of repo root - Removed unused CONFIG imports - All tests now pass without creating lingering files in repo root
- Remove MockSink class and mock_sink fixture - Remove mock_con, mock_mem, mock_redis, and mock_entity fixtures - Remove MockCollectorEntityFunc protocol - Clean up unused imports (asyncio, DragonLauncher, JobEntity) - Improves pylint score from 9.56 to 9.67
al-rigazzi
added
area: api
MattToast
requested changes
Aug 13, 2025
Co-authored-by: Matt Drozt <[email protected]>
Implement the following improvements from PR CrayLabs#789 code review: 1. Fix import style: Move shutil import to module level in test_controller_metadata_usage.py - Relocate shutil import from method to top-level imports per Python best practices 2. Remove unused JobEntity code: Complete cleanup of JobEntity ecosystem - Remove JobEntity class and _JobKey class from job.py - Remove JobEntity imports and isinstance checks from jobmanager.py - Simplify Job type annotations to use actual SmartSim entities only - Eliminate telemetry-related legacy code that's no longer needed 3. Enhance CONFIG with Path objects: Improve type safety for directory paths - Update smartsim_base_dir, dragon_default_subdir, dragon_logs_subdir, metadata_subdir to return pathlib.Path objects instead of strings - Maintain backward compatibility with os.path.join and string operations - Update test expectations to validate Path object behavior All changes tested and verified: - Import style follows Python conventions - JobEntity references completely removed from codebase - Path objects provide enhanced type safety while preserving compatibility - All existing tests pass with new Path-based CONFIG properties
Address MattToast's feedback about removing run_id which was used for telemetry tracking but is no longer needed after telemetry removal. Changes: - Remove run_id field from _LaunchedManifestMetadata NamedTuple - Remove run_id parameter from LaunchedManifestBuilder constructor - Remove run_id from serialized manifest.json output - Update all test files to remove run_id parameters - Update test expectations to use timestamp for uniqueness instead The manifest system now uses timestamp for run identification instead of the UUID-based run_id, simplifying the codebase after telemetry removal.
- Remove LaunchedManifest, _LaunchedManifestMetadata, and LaunchedManifestBuilder classes - Simplify serialize.py by removing orphaned telemetry functions (80% reduction) - Update controller.py to remove LaunchedManifest dependencies and phantom method call - Clean up all test files to remove LaunchedManifest references - Delete tests/test_serialize.py as it only tested removed functionality - Maintain core Manifest class functionality for entity organization - Achieve 10.00/10 linting score across all modified files
- Restore missing _save_orchestrator() call in _launch_orchestrator_simple() - This was accidentally removed during LaunchedManifest cleanup - Fixes test_dbnode.py::test_hosts which requires checkpoint file for reconnection - Maintains 10.00/10 linting score
- Restore missing _jobs.set_db_hosts(orchestrator) call in _launch_orchestrator_simple() - This was accidentally removed during LaunchedManifest cleanup - Fixes IndexError in db_is_active() where hosts list was empty - Resolves backend ML model test failures (test_dbmodel.py, test_dbscript.py) - Database addresses now properly populated for entity launches - Maintains 10.00/10 linting score
- Add timestamp-based unique metadata directories for each launch - Import get_ts_ms helper function from utils.helpers - Modify ensemble and model metadata directory paths to include launch timestamp - Ensures each experiment launch gets unique metadata directories - Fixes test_output_files.py::test_mutated_model_output - Prevents output file overwrites when same model is run multiple times - Historical output files now properly preserved across multiple runs - Maintains 10.00/10 linting score
- Move TStepLaunchMetaData type definition from serialize.py to controller_utils.py - Remove unused smartsim/_core/utils/serialize.py file entirely - Add pathlib.Path import to controller_utils.py for type definition - Remove TYPE_CHECKING import that was only used for the moved type - Complete final cleanup of telemetry-related serialization code - All functionality preserved and tests still pass
MattToast
requested changes
Aug 26, 2025
| step = controller._create_job_step(model, status_dir) | ||
| expected_out_path = status_dir / model.name / (model.name + ".out") | ||
| expected_err_path = status_dir / model.name / (model.name + ".err") | ||
| model.path = test_dir |
There was a problem hiding this comment.
Nit: Not really a fan of modifying attributes of globally available instances of Models just because it means that this test CAN leak state in future. If we need to set this path, can we make this Model a fixture, or use monkeypatch.setattr or something?
Comment on lines
+255
to
+270
| def test_metadata_directory_structure_with_batch_entities(self): | ||
| """Test metadata directory creation pattern with batch-like behavior""" | ||
| with tempfile.TemporaryDirectory() as temp_dir: | ||
| exp = Experiment("test_metadata_batch", exp_path=temp_dir, launcher="local") | ||
|
|
||
| # Create model and ensemble (batch settings don't work with local launcher) | ||
| model = exp.create_model( | ||
| "batch_model", | ||
| run_settings=exp.create_run_settings("echo", ["batch_hello"]), | ||
| ) | ||
|
|
||
| ensemble = exp.create_ensemble( | ||
| "batch_ensemble", | ||
| run_settings=exp.create_run_settings("echo", ["batch_world"]), | ||
| replicas=2, | ||
| ) |
There was a problem hiding this comment.
Just verifying this is intended: It looks like this test is supposed to be launching a model/ensemble with batch settings, but the local launcher is being used and no batch settings were assign to either
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR removes telemetry collection, SmartDashboard functionality, and indirect entrypoint capabilities from SmartSim. Additionally, it significantly improves the codebase's maintainability by centralizing directory path management through the CONFIG system and eliminating hardcoded
.smartsimdirectory references throughout the codebase.Changes Made
🗑️ Removed Features
🏗️ Configuration System Improvements
.smartsimdirectory references with CONFIG properties📁 New CONFIG Properties
📂 Directory Structure
The new hierarchical structure provides better organization:
Files Modified
Core Configuration
smartsim/_core/config/config.py- Enhanced with new directory properties and hierarchical structureCore Functionality
smartsim/_core/utils/serialize.py- Updated to use CONFIG.metadata_subdirsmartsim/_core/control/manifest.py- Updated to use CONFIG.metadata_subdirTest Files (15+ files updated)
High Priority:
tests/test_symlinking.py- Updated hardcoded paths to use CONFIG propertiestests/test_manifest_metadata_directories.py- Updated to use CONFIG.metadata_subdirtests/test_metadata_integration.py- Updated to use CONFIG propertiesMedium Priority:
tests/test_controller.py- Updated to use CONFIG.metadata_subdirtests/test_output_files.py- Updated comments to reference CONFIGDragon Test Files:
tests/test_dragon_step.py- Updated to use CONFIG.dragon_logs_subdirtests/test_dragon_launcher.py- Updated to use CONFIG.dragon_logs_subdirtests/test_dragon_client.py- Updated to use CONFIG.dragon_logs_subdirtests/test_dragon_run_policy.py- Updated to use CONFIG.dragon_logs_subdirBenefits
🔧 Improved Maintainability
🧪 Better Testing
🏗️ Enhanced Modularity
Technical Details
CONFIG System Architecture
The new CONFIG properties follow a hierarchical dependency model:
smartsim_base_diris the foundation (.smartsim)dragon_default_subdirbuilds on base ({base}/dragon)dragon_logs_subdirbuilds on dragon ({dragon}/logs)metadata_subdirbuilds on base ({base}/metadata)This ensures that changing the base directory automatically propagates to all subdirectories, and changing the dragon directory affects the dragon logs directory.
Path Management Implementation
All directory references now use f-string formatting with CONFIG properties:
Test Improvements
test_batch_symlinkandtest_symlinkfor better test clarityTesting
Migration Notes
For Developers:
CONFIG.metadata_subdirinstead of hardcoding ".smartsim/metadata"CONFIG.dragon_logs_subdirinstead of hardcoding ".smartsim/logs"CONFIG.smartsim_base_dirfor base directory referencesFor Users:
Backward Compatibility
Summary
This PR significantly improves SmartSim's codebase quality by:
The changes result in a cleaner, more maintainable codebase with improved consistency, better separation of concerns, and a solid foundation for future development.