Node crashes when non-important plugin installed by Reckless fails

[ShahanaFarooqui]

Core lightning node crashes when a non-important plugin fails to run, despite the documentation indicating that only important plugins should cause the daemon to crash on failure.

Expected Behavior

• According to the documentation, non-important plugins should not cause lightningd to crash.
• When a plugin is installed via reckless (which does not mark plugins as important), the failure of that plugin should be logged but should not cause the entire daemon to crash.

Actual Behavior

When the backup plugin installed via reckless fails to run, it causes the entire Core Lightning node to crash.

Steps to Reproduce

• Install backup plugin using reckless: lightning-cli reckless install "backup"
• lightningd crashes and the command fails with error lightning-cli: reading response: socket closed

Proposed Solution

• lightningd should distinguish between important and non-important plugins and handle their failures accordingly
• reckless plugin could add another rpc as listfailed with metadata like plugin_name, plugin_url, last_failed_date, failure_reason, etc.

[endothermicdev]

This is expected behavior on both counts.

Before the backup plugin can be used it has to be initialized once. The following command will create /mnt/external/location/file.sql as backup file and reference it in backup.lock in the lightning directory that stores the internal state, and which makes sure no two instances are using the same backup. (Make sure to stop your Lightning node before running this command)

poetry run ./backup-cli init --lightning-dir ~/.lightning/bitcoin file:///mnt/external/location/file.bkp

If you try to install it on a live node, it should immediately stop the node when it notices it doesn't have a synced backup (or that it's out of date.) The point is to prevent you from updating the CLN database when you can't sync the backup. This isn't done through the important mechanism, but rather through killing the lightningd process directly (by identifying the parent PID.)

At any rate, reckless installs backup successfully, and the plugin does what it's designed to do successfully. If we want the plugin to initialize a new database snapshot while the node is running, I think that's an issue better raised in https://github.com/lightningd/plugins. Otherwise we would have to disconnect all peers and basically take the node into an offline state while installing this plugin. Even then, I'm not sure we could ensure no database operations would take place in the interim.