Stacks node stuck in restart loop with error: FATAL: called a nakamoto function outside of epoch 3

[crypto-billy]

Describe the bug

I'm running a mainnet stacks node (docker image: blockstack/stacks-blockchain:3.1.0.0.3, bitcoin v28.0), the node ran into a syncing issue recently:
Burnchain block processing stops due to missing affirmed anchor stacks block hash f3e40935b65f5912a061dbbba271ee1fd3343728e3e22d237af89cd49f804980

The node is currently stuck and not able to continue syncing, has any one ran into this issue?
Any help would be appreciated, I'm able to provide my node config / full error trace if needed, thanks in advance!

Stacks node stuck in restart loop with error: FATAL: called a nakamoto function outside of epoch 3
Full error trace below:

ERRO [1738850970.869723] [testnet/stacks-node/src/main.rs:255] [chains-coordinator-0.0.0.0:20443] 
Process abort due to thread panic: panicked at /build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:314:5:
FATAL: called a nakamoto function outside of epoch 3
ERRO [1738850970.923348] [testnet/stacks-node/src/main.rs:257] [chains-coordinator-0.0.0.0:20443] Panic backtrace:    0: stacks_node::main::{{closure}}
             at build/testnet/stacks-node/src/main.rs:256:18
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/alloc/src/boxed.rs:1986:9
      std::panicking::rust_panic_with_hook
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:809:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:667:13
   3: std::sys::backtrace::__rust_end_short_backtrace
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/sys/backtrace.rs:170:18
   4: rust_begin_unwind
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:665:5
   5: core::panicking::panic_fmt
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/panicking.rs:76:14
   6: blockstack_lib::chainstate::nakamoto::coordinator::get_nakamoto_reward_cycle_info
      blockstack_lib::chainstate::nakamoto::coordinator::<impl blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>>::get_nakamoto_reward_cycle_info
             at build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:988:9
      blockstack_lib::chainstate::nakamoto::coordinator::<impl blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>>::handle_new_nakamoto_burnchain_block
             at build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:1139:21
   7: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>::handle_new_burnchain_block
             at build/stackslib/src/chainstate/coordinator/mod.rs:2436:9
      blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,blockstack_lib::chainstate::coordinator::comm::ArcCounterCoordinatorNotices,blockstack_lib::chainstate::coordinator::OnChainRewardSetProvider<T>,CE,FE,B>::handle_comms_epoch2
             at build/stackslib/src/chainstate/coordinator/mod.rs:612:19
   8: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,blockstack_lib::chainstate::coordinator::comm::ArcCounterCoordinatorNotices,blockstack_lib::chainstate::coordinator::OnChainRewardSetProvider<T>,CE,FE,B>::run
             at build/stackslib/src/chainstate/coordinator/mod.rs:578:21
   9: stacks_node::run_loop::neon::RunLoop::spawn_chains_coordinator::{{closure}}
             at build/testnet/stacks-node/src/run_loop/neon.rs:661:17
      std::sys::backtrace::__rust_begin_short_backtrace
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/sys/backtrace.rs:154:18
  10: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/thread/mod.rs:561:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/panic/unwind_safe.rs:272:9
      std::panicking::try::do_call
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:557:40
      std::panicking::try
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panicking.rs:520:19
      std::panic::catch_unwind
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/panic.rs:358:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/thread/mod.rs:559:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/core/src/ops/function.rs:250:5
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/alloc/src/boxed.rs:1972:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/alloc/src/boxed.rs:1972:9
      std::sys::pal::unix::thread::Thread::new::thread_start
             at rustc/e71f9a9a98b0faf423844bf0ba7438f29dc27d58/library/std/src/sys/pal/unix/thread.rs:105:17
  12: <unknown>
  13: __clone

Dumping core for pid 1

Steps To Reproduce

Expected behavior
Node to start syncing

Environment (please complete the following information):
Running with these docker images:

blockstack/stacks-blockchain-api:8.4.0
blockstack/stacks-blockchain:3.1.0.0.5

Additional context

If you are encountering an issue with a smart contract, please include the smart contract code
that demonstrates the issue.

[aldur]

Thank you! #5735 should fix your issue and it will be included in the next release.