Skip to content

Conversation

@BartekCupial
Copy link

It turns out that the game is somehow different based on local time https://nethackwiki.com/wiki/Time. For reproducibility and evaluation reasons we shouldn't allow this changes. For example my agent score dropped from 5k to 3k just because it was full moon.

I turned off all four such cases. moon phases, friday 13th, midnight, and night.

@BartekCupial BartekCupial changed the title turn off game specific behaviors based on time Turn off game specific behaviors based on time May 24, 2024
@heiner
Copy link
Collaborator

heiner commented May 24, 2024

Not a fan of just disabling it -- If people feel they want to have this part reproducible but setting the time to something static isn't possible in their setup (understandable on both counts) we'd need to add a hook to optionally override getlt().

But also if the goal is to beat the real NetHack we should perhaps not dumb it down too much.

@Miffyli
Copy link

Miffyli commented Jun 22, 2024

Randomly pitching in (saw that NLE is getting some love again :) ).

Agree with both here. The reliance on time is indeed very painful for running any experiments, and it is easy to miss unless you go deeper into Nethack functionality. This was a painful lesson during the NLE competition 😅

But also completely replacing it won't cut it, as it can be unexpected to people who know the logic.

I would prefer informing users about this logic more in docs. Additionally, if "completing Nethack" is baked into Gym/Gymnsium-like environments, the environment should specify the same game-time with a hook heiner suggested.

@StephenOman StephenOman added the enhancement New feature or request label Jun 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants