Illegal Realm Internal Wait on Event in an Active Message Handler

[eddy16112]

Here is the stack, I get this issue when running flecsi dependent partition branch.
#0 0x00007ffff484c1f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff484d8e8 in __GI_abort () at abort.c:90
#2 0x00007ffff4845266 in __assert_fail_base (fmt=0x7ffff4997e68 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7ffff6547b4e "thread->scheduler != 0", file=file@entry=0x7ffff6546ec8 "/home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/threads.inl", line=line@entry=236,
function=function@entry=0x7ffff6548d20 <void Realm::Thread::wait_for_conditionRealm::EventTriggeredCondition(Realm::EventTriggeredCondition const&, bool&)::PRETTY_FUNCTION> "static void Realm::Thread::wait_for_condition(const CONDTYPE&, bool&) [with CONDTYPE = Realm::EventTriggeredCondition]")
at assert.c:92
#3 0x00007ffff4845312 in __GI___assert_fail (assertion=assertion@entry=0x7ffff6547b4e "thread->scheduler != 0", file=file@entry=0x7ffff6546ec8 "/home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/threads.inl", line=line@entry=236,
function=function@entry=0x7ffff6548d20 <void Realm::Thread::wait_for_conditionRealm::EventTriggeredCondition(Realm::EventTriggeredCondition const&, bool&)::PRETTY_FUNCTION> "static void Realm::Thread::wait_for_condition(const CONDTYPE&, bool&) [with CONDTYPE = Realm::EventTriggeredCondition]")
at assert.c:101
#4 0x00007ffff63332e0 in Realm::Thread::wait_for_conditionRealm::EventTriggeredCondition (cond=..., poisoned=@0x7fff29cbe9ff: false) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/threads.inl:236
#5 0x00007ffff633114d in Realm::Event::wait_faultaware (this=this@entry=0x7fff29cbea38, poisoned=poisoned@entry=@0x7fff29cbe9ff: false) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/event_impl.cc:282
#6 0x00007ffff633133c in Realm::Event::wait (this=0x7fff29cbea38) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/event_impl.cc:243
#7 0x00007ffff78b8bf9 in Realm::SparsityMapPublicImpl<1, long long>::get_entries (this=0x7fff240359e0) at /home/wwu12/flecsi/flecsi-third-party-debug/include/realm/sparsity.inl:71
#8 0x00007ffff78b9a72 in Realm::IndexSpaceIterator<1, long long>::reset (this=0x7fff24031188, _space=...) at /home/wwu12/flecsi/flecsi-third-party-debug/include/realm/indexspace.inl:1394
#9 0x00007ffff5f00ece in Realm::TransferIteratorIndexSpace<1, long long>::deserialize_newRealm::Serialization::FixedBufferDeserializer (deserializer=...) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/transfer/transfer.cc:1066
#10 0x00007ffff5f01044 in Realm::Serialization::PolymorphicSerdezSubclass<Realm::TransferIterator, Realm::TransferIteratorIndexSpace<1, long long> >::deserialize_new (this=, deserializer=...) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/serialize.inl:738
#11 0x00007ffff5ee4d8b in Realm::Serialization::PolymorphicSerdezHelperRealm::TransferIterator::deserialize_new (deserializer=...) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/serialize.inl:679
#12 0x00007ffff5edf0e9 in deserialize_newRealm::Serialization::FixedBufferDeserializer (deserializer=...) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/transfer/transfer.h:97
#13 Realm::XferDesCreateMessage::handle_request (args=..., msgdata=, msglen=) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/transfer/channel.cc:3270
#14 0x00007ffff63a7ccd in IncomingMediumMessage<Realm::XferDesCreateMessage::RequestArgs, 180, &Realm::XferDesCreateMessage::handle_request, 12>::run_handler (this=0x7fff1c001140) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/activemsg.h:357
#15 0x00007ffff63cca76 in IncomingMessageManager::handler_thread_loop (this=0xd9e350) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/activemsg.cc:1173
#16 0x00007ffff63d3de1 in Realm::Thread::thread_entry_wrapper<IncomingMessageManager, &IncomingMessageManager::handler_thread_loop> (obj=) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/threads.inl:131
#17 0x00007ffff63c12ca in Realm::KernelThread::pthread_entry (data=0xd9e4e0) at /home/wwu12/flecsi/flecsi-third-party/legion/runtime/realm/threads.cc:700
#18 0x00007ffff5680e25 in start_thread (arg=0x7fff29cc6700) at pthread_create.c:308
#19 0x00007ffff490f34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

[eddy16112]

It is using Legion master branch

[eddy16112]

To reproduce the problem without installing the entire FLECSI, please ping me in slack and I will send you the pwd to the machine I am using

[streichler]

I'll take a look. I fixed another bug right in this area recently, so I may be able to find it by code inspection. If not, I'll ask for the login info.

[eddy16112]

Thanks for letting me know. I just saw the fix for the events in active messages. The built-in Legion of FleCSI does not update to that commit. I just updated the Legion and issue is gone now. I will leave the issue open for a week, and keep you updated if it happens again,

[streichler]

@eddy16112 It's been almost a week and I'm impatient... :) Any new sightings of this issue?

[eddy16112]

@streichler I did not see this issue any more. I will close it. Thanks!