feat: Introduce NePS-Spaces #248
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced tests for various aspects of the NEPS space, including: - `test_search_space__nos_like.py`: Tests for the resolution of a complex pipeline with unary, binary, and ternary operations. - `test_search_space__recursion.py`: Validates recursive operations within the pipeline, ensuring correct factor propagation. - `test_search_space__resampled.py`: Tests for resampling behavior in both float and integer pipelines, ensuring shared and unique values are handled correctly. - `test_search_space__reuse_arch_elements.py`: Verifies the reuse of architectural elements in pipelines, ensuring correct operation and configuration string generation. - `utils.py`: Added utility functions to generate possible configuration strings for pipelines, aiding in testing and validation.
…odule - Changed all instances of `space` to `neps_space` in test files to reflect the new import structure. - Updated class definitions to inherit from `neps_space.Pipeline` instead of `space.Pipeline`. - Modified the creation of operations and categorical choices to use `neps_space.Operation` and `neps_space.Categorical`. - Adjusted the resolution and conversion functions to utilize the new `neps_space` methods. - Ensured consistency across all test cases for the new structure and functionality.
- Added a new module `bracket_optimizer.py` that implements a bracket-based optimization strategy for multi-fidelity configurations in NePS spaces. - Introduced the `_BracketOptimizer` class to handle the optimization process, including sampling configurations from a prior band and managing different fidelity levels. - Implemented various bracket types such as successive halving, hyperband, ASHA, and async hyperband. - Updated `priorband.py` to utilize the new sampling strategies and refactored the prior sampling logic. - Created a new `sampling.py` module to define various samplers, including `OnlyPredefinedValuesSampler`, `RandomSampler`, `PriorOrFallbackSampler`, and mutation-based samplers. - Modified existing tests to accommodate the new optimizer and sampling strategies, ensuring compatibility with the updated structure. - Refactored import paths in test files to align with the new module organization.
- Updated import statements to use `neps.space.neps_spaces.parameters` for all relevant classes and functions in test files. - Refactored test classes and methods to replace direct usage of `neps_space` with the new parameters module, ensuring consistency across the test suite. - Adjusted the instantiation of various operations and categorical choices to align with the new structure. - Ensured that all references to operations, floats, integers, and categorical choices are correctly sourced from the updated parameters module.
- Updated imports in test files to directly import necessary classes from neps.space.neps_spaces.parameters. - Refactored class definitions to inherit directly from Pipeline and other relevant classes. - Replaced instances of parameter class references with direct class references for improved readability. - Ensured consistency across test files by applying similar refactoring patterns.
…pipeline adjustments
- Updated imports for optimizers in test_neps_integration.py to use the new neps.optimizers module. - Replaced references to old RandomSearch and ComplexRandomSearch algorithms with their new counterparts from neps_algorithms. - Adjusted the optimizer imports in test_neps_integration_priorband__max_cost.py and test_neps_integration_priorband__max_evals.py to align with the new structure. - Ensured all instances of priorband optimizers are updated to use the new neps_algorithms module.
…eration classes to improve clarity and consistency
…arity and consistency - Updated docstrings in SamplingResolutionContext to use Args and Returns sections for better readability. - Changed parameter names from `prior_index` to `prior` in Categorical class to enhance clarity. - Introduced a new function `convert_confidence_level` to handle string to ConfidenceLevel conversion. - Updated tests to reflect changes in parameter names from `prior_index` to `prior`. - Enhanced docstrings across various classes and methods in the sampling module for consistency.
… and convert_confidence_level function
…es and enhancing links to related topics
…ing clarity, and improving formatting in code examples
… HPO classes - Updated the image segmentation example to utilize HPO_Float and HPO_Integer for hyperparameter definitions. - Refactored test cases to replace old parameter classes with HPO variants, ensuring compatibility with the new hyperparameter optimization framework. - Added log sampling for NePS-Integer - Adjusted imports in test files to reflect changes in the neps library structure. - Enhanced the configuration encoder tests to validate the new HPO parameter types. - Ensured all relevant tests are passing with the updated hyperparameter definitions.
…md file - Corrected links in getting_started.md, evaluate_pipeline.md, and neps_run.md to point to neps_spaces.md. - Updated navigation in mkdocs.yml to reflect the removal of pipeline_space.md. - Enhanced clarity in algorithms.py and grid.py by updating parameter references. - Added validation in parameters.py to ensure prior_confidence is set when prior is defined.
…nteger, Operation, and Resampled classes
…atibility checks and conversions
…n neps.run() and enhance fidelity parameter explanation
…with pre-commit changes
…into neps-spaces-updates
…prior method in PipelineSpace
…ter readability and maintainability
…cking and improve readability
…thods to avoid using a cached resolved object instead of resolving when they are actually different
…y when required and improve DataFrame handling
…ts for consistent trace text generation
…s for better organization
… metrics functionality - Introduced tests for converting between classic SearchSpace and NePS PipelineSpace, ensuring attributes and properties are preserved. - Implemented tests for algorithm compatibility with both NePS-only and classic algorithms. - Added tests for trajectory and metrics functionality, including handling of fidelity parameters and cumulative metrics tracking. - Included error handling tests for failed evaluations and validation of metric values. - Verified file format and structure for trajectory and best config outputs. - Ensured that NePS can revisit previous runs and update trajectories correctly.
…cripts - Deleted `priors_test.ipynb` and `warmstarting.py` as they were no longer needed.
…egration tests for hyperparameter optimization with fidelity
-> Add trial validation and import functionality - Introduced `TrialValidationError` exception for handling trial configuration validation errors. - Created a new module `normalization.py` for normalizing imported configurations. - Implemented `import_trials` method in various optimizers (e.g., `BayesianOptimization`, `BracketOptimizer`, `GridSearch`, `IFBO`, `PriMO`, `RandomSearch`) to handle external evaluations. - Added validation functions in `validation.py` to ensure imported results and configurations meet expected criteria. - Enhanced `Trial` class to support loading from external data with a new `load_from_dict` method. - Updated `BudgetInfo` and `WorkerSettings` to allow for float values in `fidelities_to_spend`. - Added utility functions in `trial_io.py` for loading trials from pickle files. - Created an example script `example_import_trials.py` to demonstrate the import trials functionality.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request updates the documentation to standardize and modernize how search spaces are defined and referenced in NePS. The main change is moving from dictionary-based search space definitions to using classes that inherit from
neps.PipelineSpace.Migration to class-based search space definitions:
docs/getting_started.md,docs/index.md, anddocs/reference/neps_run.mdto use a class inheriting fromneps.PipelineSpaceinstead of a dictionary for defining the search space. This includes updating parameter definitions to useneps.Float,neps.Integer, andneps.Categoricalwith new argument names likemin_valueandmax_value. [1] [2] [3]Reference and terminology updates:
neps.run(evaluate_pipeline, ExampleSpace())instead of passing a dictionary. [1] [2] [3]Minor formatting and clarity improvements: