Management for custom node collections

[joepavitt]

Also covers areas: Version Control & Centralized Management

Background

We've heard from many customers that it is common to build custom nodes for their specific use cases. These custom nodes generally come in two forms:

• Subflows: User can select a flow within the Node-RED editor and create a "subflow", which becomes a re-usable asset in that particular editor. The user can then save this to the Team Library in FlowFuse, and import it to other Instances within FlowFuse too.
• Custom nodes: A developer has coded a custom Node-RED node and published it either to the public NPM registry, or to a private NPM registry made available only to the respective team/business.

Current Offerings

Team Library

FlowFuse offers the Team Library as a centralised repository for common flow.json samples or .js function nodes that can be imported to any Instances in the team.

Pain Points:

• Updating a flow.json or .js file is a manual process, users must manually re-import and replace each instance of any imported contents from the Team Library.
• There is no concept of version control here, once a file is overriden, the previous iterations are lost.

Private NPM Registry

In FlowFuse it is possible to configure external, private NPM registries. This makes any present Node-RED nodes in that registry available in the Palette Manager

Pain Points

• User needs to setup, host and configure their own third-party npm registry
• Requires users to develop/code their own nodes. This is not a solution that covers subflows.

Proposal

• We have version control built in via the palette manager, where used can update to the latest version for a given node.
• Subflows are essentially a single node, but just configured via a flow.json, rather than a repository with an html, js file, etc.
• Can we build in tooling that permits users to publish subflows to a private NPM repository?
• Can we host a private NPM repository (per team, or per platform?) to make the onboarding easier and prevent the need for customers to configure and host their own.

Customers

Which customers would this be available to

Team/Enterprise

Which customers have requested this?

• https://app-eu1.hubspot.com/contacts/26586079/record/0-2/18782320332

[joepavitt]

Relevant issue: #3666
Relevant issue: #3734

[joepavitt]

Notes from brainstorming session:

Can we host a private NPM repository (per team, or per platform?) to make the onboarding easier and prevent the need for customers to configure and host their own.

@hardillb - there are open source containers for the registry, but not sure we can do this with a single registry to provide to multiple teams. - https://verdaccio.org/

• Key here is that we need compartmentalising the registries/collections.
• Ideally, we just run one, but scope the projects to teams - far less maintenance.
• Needs technical exploration on the "what is possible?"
• @hardillb - I don't like the idea of running registry per team.
• It's going t cost us a lot to run this, as it's new containers
  • @joepavitt - this will be scoped to Enterprise
• How do users publish to this?
  • Joe: I'd be expecting this was an npm publish
  • Scoping @teamId/package-name
• registry.flowfuse.com vs. registry.team.flowfuse.com complexity/overhead of management too
• @knolleary can this be an opt-in feature? Especially in case whereby the registries needs to be per team to save costs

Actions

• PoC on authentication/access control over NPM registry, verify if a single registry is possible, and what constraints exist

[hardillb]

We can run a single npm registry and divide access based on package scope.

Will need to create a user with a fixed password (to be verified via HTTP request to FF App)

[joepavitt]

Will need to create a user with a fixed password (to be verified via HTTP request to FF App)

As in, each FlowFuse user will need a specific npm user?

[hardillb]

Each team will need a password, not each user.

But that does raise some interesting security thoughts (how to revoke access if user is removed from a team)

[hardillb]

PoC outcome

• A single npm registry can cater for multiple user by partitioning using package scopes
• Authentication at FF Team level is possible for MVP (doing at a individual team owner level [I propose that only team owners should be able to push new packages] is a LOT more complicated and much worse UX)
• generating catalogue.json files from the repo should be possible while the number of packages remains small, but there is a question about how to do it efficiently at scale (will need to effectively pull ALL the package in one go, then filter by scope into separate catalogue files per team).

I have raised sub issues for tasks required to implement this

• Write a Verdaccio authentication plugin #5164
• Create a hosting plan for a npm registry in FFC #5165
• Update nr-launcher or instance settings API to add details to .npmrc #5166