Skip to content

RNG leads to repeated random numbers or non-reproducible results #80

@tautomer

Description

@tautomer

Problem:

  • When a seed is provided, the package resets the random number generator (RNG) states multiple times within a single campaign:

    • The optimizer, model, designer, and DOE components each reset RNG states independently.
    • During a normal workflow (without explicitly defining the optimizer/designer), random seed reset happens when
      • the optimizer is initialized (torch and numpy)
      • the designer is initialized (torch only)
        • if DOE is used (numpy only)
      • every iteration when each model is instantiated (torch and numpy)
  • When no seed is provided, there is no RNG control at all, resulting in no reproducibility.

Impact:

Repeated RNG resets cause correlated and repeated noise that biases results. In the model fitting stage, the model and scipy/Adam optimizer will always start from the same initial conditions. Additionally, global seed control is fragile, as unexpected random draws between steps break reproducibility unless perfectly replicated.

Proposed Fix:

  • Centralize RNG state management to avoid multiple independent resets.
  • Ensure the simulator noise RNG is initialized once per campaign and not reset every iteration.
  • Provide consistent RNG control whether or not a seed is specified, enabling reproducibility.
  • Refactor the code to use a single RNG state or RNG generator instance passed explicitly to components that require randomness.
  • Add a package level toggle to switch back to current behavior for backward compatibility.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions