Invalid remove version state bug

[magnatelee]

Running the regent miniaero with a release binary on two nodes raises the following assertion failure:

terra: /home/wclee/Workspace/legion/runtime/legion/garbage_collection.cc:665: bool LegionRuntime::HighLevel::DistributedCollectable::update_state(bool&, bool&, bool&, bool&, bool&): Assertion `false' failed.

However, this bug is not reproducible with a debug binary, the binding library needs to be manually compiled with options "CC_FLAGS='-g' DEBUG=0" passed.

Here is the stacktrace:
#0 0x00007fb063f6c9bd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fb063f6c854 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2 0x00007fb063aa2ffa in Realm::realm_freeze (signal=6) at /home/wclee/Workspace/legion/runtime/realm/runtime_impl.cc:75
#3
#4 0x00007fb063ee1bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5 0x00007fb063ee4fc8 in __GI_abort () at abort.c:89
#6 0x00007fb063edaa76 in __assert_fail_base (fmt=0x7fb06402c370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7fb063bd71c9 "false", file=file@entry=0x7fb063bc9bd8 "/home/wclee/Workspace/legion/runtime/legion/garbage_collection.cc", line=line@entry=665,

function=function@entry=0x7fb063bc9c40 <LegionRuntime::HighLevel::DistributedCollectable::update_state(bool&, bool&, bool&, bool&, bool&)::__PRETTY_FUNCTION__> "bool LegionRuntime::HighLevel::DistributedCollectable::update_state(bool&, bool&, bool&, bool&, bool&)") at assert.c:92

#7 0x00007fb063edab22 in __GI___assert_fail (assertion=0x7fb063bd71c9 "false", file=0x7fb063bc9bd8 "/home/wclee/Workspace/legion/runtime/legion/garbage_collection.cc", line=665,

function=0x7fb063bc9c40 <LegionRuntime::HighLevel::DistributedCollectable::update_state(bool&, bool&, bool&, bool&, bool&)::__PRETTY_FUNCTION__> "bool LegionRuntime::HighLevel::DistributedCollectable::update_state(bool&, bool&, bool&, bool&, bool&)") at assert.c:101

#8 0x00007fb063a9378f in LegionRuntime::HighLevel::DistributedCollectable::update_state (this=this@entry=0x7fb010adc8a0, need_activate=@0x7fb0101fff8b: false, need_validate=@0x7fb0101fff8c: false, need_invalidate=@0x7fb0101fff8d: false, need_deactivate=@0x7fb0101fff8e: false, do_deletion=@0x7fb0101fff8f: false)

at /home/wclee/Workspace/legion/runtime/legion/garbage_collection.cc:665

#9 0x00007fb063a938ea in LegionRuntime::HighLevel::DistributedCollectable::add_gc_reference (this=0x7fb010adc8a0, cnt=cnt@entry=1) at /home/wclee/Workspace/legion/runtime/legion/garbage_collection.cc:115
#10 0x00007fb0639cc7fe in add_nested_gc_ref (source=, cnt=1, this=) at /home/wclee/Workspace/legion/runtime/legion/garbage_collection.h:343
#11 LegionRuntime::HighLevel::VersionState::notify_active (this=0x7fb010b0ec60) at /home/wclee/Workspace/legion/runtime/legion/legion_analysis.cc:5205
#12 0x00007fb063a9394b in LegionRuntime::HighLevel::DistributedCollectable::add_gc_reference (this=0x7fb010b0ec60, cnt=cnt@entry=1) at /home/wclee/Workspace/legion/runtime/legion/garbage_collection.cc:101
#13 0x00007fb0639d86db in add_base_gc_ref (source=LegionRuntime::HighLevel::PHYSICAL_STATE_REF, cnt=1, this=) at /home/wclee/Workspace/legion/runtime/legion/garbage_collection.h:332
#14 LegionRuntime::HighLevel::PhysicalState::add_version_state (this=, state=0x7fb010b0ec60, state_mask=...) at /home/wclee/Workspace/legion/runtime/legion/legion_analysis.cc:4008
#15 0x00007fb0639d9d72 in LegionRuntime::HighLevel::VersionInfo::unpack_node_info (this=this@entry=0x7fb010462480, node=0x7fb010a99ea0, ctx=ctx@entry=0, derez=..., source=source@entry=0) at /home/wclee/Workspace/legion/runtime/legion/legion_analysis.cc:1061
#16 0x00007fb0639d9f88 in LegionRuntime::HighLevel::VersionInfo::unpack_buffer (this=0x7fb010462480, forest=0x7fb010204a20, ctx=0) at /home/wclee/Workspace/legion/runtime/legion/legion_analysis.cc:918
#17 0x00007fb0639ebcca in LegionRuntime::HighLevel::VersionInfo::make_local (this=0x7fb010462480, preconditions=std::set with 0 elements, forest=, ctx=) at /home/wclee/Workspace/legion/runtime/legion/legion_analysis.cc:672
#18 0x00007fb063966847 in LegionRuntime::HighLevel::SliceTask::trigger_remote_state_analysis (this=0x7fb0104615c0, ready_event=...) at /home/wclee/Workspace/legion/runtime/legion/legion_tasks.cc:9896
#19 0x00007fb063a7e0fe in LegionRuntime::HighLevel::Runtime::high_level_runtime_task (args=0x7fb010aa04a0, arglen=, p=...) at /home/wclee/Workspace/legion/runtime/legion/runtime.cc:16019
#20 0x00007fb063ae54a6 in Realm::Task::execute_on_processor (this=0x7fb010b5e420, p=...) at /home/wclee/Workspace/legion/runtime/realm/tasks.cc:80
#21 0x00007fb063ae55a5 in Realm::UserThreadTaskScheduler::execute_task (this=, task=) at /home/wclee/Workspace/legion/runtime/realm/tasks.cc:887
#22 0x00007fb063ae6538 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x3dd9000) at /home/wclee/Workspace/legion/runtime/realm/tasks.cc:448
#23 0x00007fb063ad6501 in Realm::UserThread::uthread_entry () at /home/wclee/Workspace/legion/runtime/realm/threads.cc:740
#24 0x00007fb063ef47a0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#25 0x0000000000000000 in ?? ()

[lightsighter]

Pushed a fix for this bug to the master branch. Assigning Wonchan to confirm that it is fixed and close the bug.

[magnatelee]

This issue seems gone for now. Reopen when it exhibits again.