Refactor: Modular architecture and Ray integration for distributed feature engineering

[taha2samy]

1. Monolithic to Modular Package Refactor

Replaced the single, hard-to-maintain bigfeat_base.py with a specialized package structure. This separation of concerns allows for easier debugging and independent scaling of generation and selection modules.

Architecture Evolution:

# BEFORE (Monolith)                # AFTER (Modular Package)
BigFeat-master/                    BigFeat-master/
└── bigfeat/                       └── bigfeat/
    └── bigfeat_base.py (1.5k LOC)     ├── __init__.py
                                       ├── base.py (Orchestrator)
                                       ├── generator.py (Logic)
                                       ├── importance.py (Models)
                                       ├── selection.py (Filtering)
                                       └── distributed_tasks.py (Ray)

2. Distributed Architecture Integration (Ray)

The legacy model executed iterations sequentially. I have integrated Ray to enable parallel execution of feature crossing and importance scoring across multiple CPU cores or remote clusters.

Code Shift:

• Legacy: for i in range(gen_size): self.feat_with_depth(...) (Blocking)
• New: remote_generate_batch.remote(...) (Asynchronous/Distributed)
• Connectivity: Added config.py to support ray:// addresses for Kubernetes or local auto-scaling.

3. Optimized Memory Management (Shared Object Store)

Passing large NumPy matrices directly to class methods in a distributed environment causes massive memory overhead. I implemented Ray Object Store references.

Implementation Hook:

# Instead of passing the actual matrix X, we store it once:
x_ref = ray.put(x_scaled) 

# Workers now only receive a lightweight 20-byte reference (x_ref)
# to access the shared memory, preventing OOM (Out of Memory) crashes.

4. Conversion to Stateless Pure Functions

Refactored logic out of the BigFeat class into standalone Pure Functions in generator.py. This was critical because class methods carry the self context, which is often too heavy to "pickle" and send over the network.

Logic Refactoring:

# BEFORE (Class Bound)
def feat_with_depth(self, X, ...):
    op = self.rng.choice(self.operators) # Bound to instance state

# AFTER (Stateless/Pure)
def feat_with_depth(X, ..., rng, operators): 
    # Can be easily serialized and sent to any remote worker

5. Distributed Importance Logic & LightGBM Fix

Importance calculations now happen "close to the data" on remote workers. I also identified and fixed a silent bug in the original LightGBM implementation where parameters were being reset.

The Fix:

• Bug: The original code had param = {} immediately following parameter definitions, erasing num_leaves and objective.
• Correction: Proper parameter mapping is now maintained for both classification and regression tasks within importance.py.

6. Numerical Stability (Epsilon Patch)

To prevent the system from crashing during massive feature generation, I injected an Epsilon ($1e-9$) constant into all probability and importance calculations.

Code Protection:

# Prevents ZeroDivisionError if a feature has zero gain
ig_vector = importance_sum / (importance_sum.sum() + 1e-9)

7. Parallel-Safe Randomness (Modern NumPy)

Swapped the old np.random.RandomState for the modern np.random.default_rng. In a distributed system, using the old method often leads to "collisions" (workers generating the exact same features).

Seeding Strategy:

• Implemented a dynamic seed generator: rng_seed = random_state + batch_index + iteration.
• This ensures that every parallel worker explores a different part of the feature space, maximizing entropy and feature diversity.

8. Mathematical Utility Refactoring & Modernization

Decoupled mathematical operators into local_utils.py and modernized them to ensure compatibility with modern scientific stacks.

• Operator Standardization: Refactored unary transformations including unary_cube, unary_multinv, unary_sqrtabs, and unary_logabs for better integration into the crossing logic.
• Modern SciPy Patching: Updated the mode function to include keepdims=True, preventing runtime failures and warnings in SciPy 1.11+ environments.

9. Stateful & Isolated Feature Selection

Transitioned the feature selection logic into a dedicated, stateful selection.py module.

• Pipeline Consistency: The selector object (e.g., fAnova) now preserves its fitted state within the class. This ensures that the transform phase applies the exact same statistical filtering and feature mapping determined during the fit phase on new data.

10. Unified Model Evaluation Engine

Consolidated model selection and scoring logic into a centralized evaluation.py module.

• Explicit Growth Control: Enforced explicit hyper-parameters for tree-based models (e.g., min_samples_leaf=1, max_features='sqrt'). This ensures trees are fully grown to extract high-variance, high-precision importance scores during feature generation.

11. Dynamic Load Balancing (Ray Orchestration)

Overhauled the orchestrator in base.py to support high-throughput distributed execution.

• Intelligent Batching: Implemented a load-balancing mechanism that polls ray.cluster_resources(). It dynamically scales the batch_size based on available CPU cores, eliminating idle worker time and optimizing cluster utilization.

12. Tree Path Optimization & Filtering

Optimized the tree-utility logic within tree_utils.py to handle large-scale importance scoring.

• Redundancy Filtering: Enhanced get_paths with a unique-path filter. By discarding redundant tree branches, the system drastically reduces the computational load required to aggregate split-based feature importances.

13. Experimental Testing Infrastructure (Alpha)

Introduced the project's first automated testing suite, currently in an experimental/alpha stage.

• Distributed Lifecycle: Includes conftest.py to manage the automated setup and teardown of local/mock Ray clusters.
• Operator Validation: Includes test_utils.py to rigorously verify the mathematical integrity and output consistency of each feature operator.