Remove accessory from all services from Apple Home => device not removed from Matter Server

[olavt]

I have added an Matter accessory both to Apple Home and to Open Home Foundation Matter Server and the device works fine in both ecosystems.

Then I select "Remove" in Apple Home and then "Remove from All Services".

The device is successfully removed from Apple Home, but it doesn't seem to be properly removed from Matter Server.

I get this message regarding the removed node:

{
"event": "node_event",
"data": {
"node_id": 2,
"endpoint_id": 0,
"cluster_id": 40,
"event_id": 2,
"event_number": 65552,
"priority": 1,
"timestamp": 619084,
"timestamp_type": 0,
"data": {
"fabricIndex": 2
}
}
}

Then after this I get this in the Matter Server log:

2025-01-23 22:45:38.192 (Dummy-2) CHIP_ERROR [chip.native.EM] <<5 [E:28549r with Node: <0000000000000002, 1> S:9548 M:140120936] (S) Msg Retransmission to 1:0000000000000002 failure (max retries:4)
2025-01-23 22:45:57.723 (Dummy-2) CHIP_ERROR [chip.native.EM] <<5 [E:22496i with Node: <0000000000000002, 1> S:9548 M:140120937] (S) Msg Retransmission to 1:0000000000000002 failure (max retries:4)
2025-01-23 22:46:05.764 (Dummy-2) CHIP_ERROR [chip.native.DMG] Time out! failed to receive report data from Exchange: 22496i with Node: <0000000000000002, 1>
2025-01-23 22:46:05.766 (MainThread) ERROR [matter_server.server.client_handler] [547543183152] Error while handling: read_attribute (node 2): src/app/ReadClient.cpp:723: CHIP Error 0x00000032: Timeout
2025-01-23 22:46:36.832 (Dummy-2) CHIP_ERROR [chip.native.DIS] Timeout waiting for mDNS resolution.
2025-01-23 22:46:50.822 (Dummy-2) CHIP_ERROR [chip.native.DIS] OperationalSessionSetup[1:0000000000000002]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:46:50.824 (MainThread) ERROR [matter_server.server.client_handler] [547543183152] Error while handling: read_attribute (node 2): src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:46:51.270 (Dummy-2) CHIP_ERROR [chip.native.DMG] Subscription Liveness timeout with SubscriptionID = 0xe9e83114, Peer = 01:0000000000000002
2025-01-23 22:46:51.272 (MainThread) INFO [matter_server.server.device_controller] Node:2 Subscription failed with CHIP Error 0x00000032: Timeout, resubscription attempt 0
2025-01-23 22:47:21.881 (Dummy-2) CHIP_ERROR [chip.native.DIS] Timeout waiting for mDNS resolution.
2025-01-23 22:47:35.879 (Dummy-2) CHIP_ERROR [chip.native.DIS] OperationalSessionSetup[1:0000000000000002]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:47:35.880 (Dummy-2) CHIP_ERROR [chip.native.DMG] Failed to establish CASE for re-subscription with error 'src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout'
2025-01-23 22:47:35.883 (MainThread) INFO [matter_server.server.device_controller] Node:2 Subscription failed with CHIP Error 0x00000032: Timeout, resubscription attempt 1
2025-01-23 22:47:35.884 (MainThread) ERROR [matter_server.server.client_handler] [547543183152] Error while handling: read_attribute (node 2): src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:48:15.250 (Dummy-2) CHIP_ERROR [chip.native.DIS] Timeout waiting for mDNS resolution.
2025-01-23 22:48:29.228 (Dummy-2) CHIP_ERROR [chip.native.DIS] OperationalSessionSetup[1:0000000000000002]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:48:29.229 (Dummy-2) CHIP_ERROR [chip.native.DMG] Failed to establish CASE for re-subscription with error 'src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout'
2025-01-23 22:48:29.232 (MainThread) INFO [matter_server.server.device_controller] Node:2 Subscription failed with CHIP Error 0x00000032: Timeout, resubscription attempt 2
2025-01-23 22:48:29.232 (MainThread) INFO [matter_server.server.device_controller] Node:2 Marked node as unavailable
2025-01-23 22:49:03.737 (Dummy-2) CHIP_ERROR [chip.native.DIS] Timeout waiting for mDNS resolution.
2025-01-23 22:49:17.719 (Dummy-2) CHIP_ERROR [chip.native.DIS] OperationalSessionSetup[1:0000000000000002]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:49:17.720 (Dummy-2) CHIP_ERROR [chip.native.DMG] Failed to establish CASE for re-subscription with error 'src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout'
2025-01-23 22:49:17.722 (MainThread) INFO [matter_server.server.device_controller] Node:2 Subscription failed with CHIP Error 0x00000032: Timeout, resubscription attempt 3
2025-01-23 22:49:57.204 (Dummy-2) CHIP_ERROR [chip.native.DIS] Timeout waiting for mDNS resolution.
2025-01-23 22:50:11.198 (Dummy-2) CHIP_ERROR [chip.native.DIS] OperationalSessionSetup[1:0000000000000002]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:50:11.199 (Dummy-2) CHIP_ERROR [chip.native.DMG] Failed to establish CASE for re-subscription with error 'src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout'
2025-01-23 22:50:11.201 (MainThread) INFO [matter_server.server.device_controller] Node:2 Subscription failed with CHIP Error 0x00000032: Timeout, resubscription attempt 4
2025-01-23 22:51:09.071 (Dummy-2) CHIP_ERROR [chip.native.DIS] Timeout waiting for mDNS resolution.
2025-01-23 22:51:23.056 (Dummy-2) CHIP_ERROR [chip.native.DIS] OperationalSessionSetup[1:0000000000000002]: operational discovery failed: src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout
2025-01-23 22:51:23.057 (Dummy-2) CHIP_ERROR [chip.native.DMG] Failed to establish CASE for re-subscription with error 'src/lib/address_resolve/AddressResolve_DefaultImpl.cpp:123: CHIP Error 0x00000032: Timeout'
2025-01-23 22:51:23.059 (MainThread) INFO [matter_server.server.device_controller] Node:2 Subscription failed with CHIP Error 0x00000032: Timeout, resubscription attempt 5

If I restart the Matter Server, the node I removed from Apple Home is still present, but it's "dead".

I have tested this 2 times with the same results.

[olavt]

I think the main issue here is that the Matter Server does not handle the Basic Information Cluster Leave event. I have not examined the Python code to verify this claim.

Now, the big question is if the Matter Server should clean up it's state regardless how the client would handle this event, or if it would be the client's responsibility to call the Matter Server to remove the node. One issue is that the Matter Server still assumes that the node is present and tries to re-establish subscriptions.

I tried as a workaround from my client to handle the Node Leave event and called the "remove_node" command, but that command timed out (since the node already is removed I guess).

My best guess right now is that the Matter Server should handle the event and do some cleanup. The client can also the event to do required cleanup that the client may need to do.

There are other stuff that the client may be responsible for cleaning up too.

From the Matter 1.4 specification:

The Leave event SHOULD be generated by a Node prior to permanently leaving a given Fabric, such
as when the RemoveFabric command is invoked for a given fabric, or triggered by factory reset or
some other manufacturer specific action to disable or reset the operational data in the Node. When
a Leave event is generated, it SHOULD be assumed that the fabric recorded in the event is no longer
usable, and subsequent interactions targeting that fabric will most likely fail.
Upon receipt of Leave Event on a subscription, the receiving Node MAY update other nodes in the
fabric by removing related bindings, access control list entries and other data referencing the leaving Node.

[marcelveldt]

Yeah, I'm pretty sure we didn't even implement that in the server because so far that hasnt been the goal.
Keep this issue open as a feature request so we can add it in later.

[Apollon77]

We will revisit and check this when server got migrated to matter.js