feat(POC): add run time config management service #2355

Junjiequan · 2025-11-17T15:36:18Z

Description

Motivation

Fixes

Bug fixed (#X)

Changes:

changes made

Tests included

Included for each change/fix?
Passing?

Documentation

swagger documentation updated (required for API changes)
official documentation updated

official documentation info

…inals for sync change logging

sbliven · 2025-11-25T15:21:34Z

@Junjiequan Can you add an example for how to use this?

bpedersen2

Sound good in general.

I guess a filter for certain fields will be needed in the end if we allow be-config changes, e.g. to avoid loss of all admin groups or changes to mongodb settings( I would probably keep all security-related fields as file-only).

Frontend-config should be less critical.

sbliven

I think we should discuss the core premise of this POC, which is that the configuration should be stored in the database instead of in files. While it would be convenient to be able to change the configuration on the fly, I'm worried that it will become even harder to track configuration changes. If we do it, I think there should be a clear process to "burn in" the configuration: export non-default settings to an env file, empty the database, and restart the backend with a clean configuration. Otherwise it's hard to keep track of changes.

Can you add a description of the usage to the README or docs? Even if it changes later, it would be helpful to read how the merging works. I think you said that the database is considered canon and overrides anything in the config files or the defaults.

How does this fit with /admin/config? Would this provide the merged "frontend" data?

Likewise, will calls to the configService be merged to the "backend" data? If so, we should double-check that there aren't any remaining direct calls to configuration(), as those would bypass the configService.

I guess you would add more endpoints

get the full, merged configuration
get/set individual properties (or would this just be done with PATCH application/json+merge?)
way to delete configuration changes (resetting from the files or defaults)

sbliven · 2025-11-26T15:34:28Z

src/runtime-config/runtime-config.controller.ts

+export class RuntimeConfigController {
+  constructor(private readonly runtimeConfigService: RuntimeConfigService) {}
+
+  @AllowAny()


Needs the swagger descriptors (ApiOperation, ApiParam, etc)

sbliven · 2025-11-26T15:42:38Z

src/runtime-config/dto/runtime-config.dto.ts

+    example: "frontend",
+  })
+  @IsString()
+  _id: string;


I suggest key for the unique identifier. _id should only be used for the internal mongo document id (which should not be accessible in the API).

Does it make sense to have more than "frontend" and "backend"? I would make this an enum. Or are you thinking there might be use cases where other microservices store most of their configs on the scicat backend?

_id should only be used for the internal mongo document id (which should not be accessible in the API).

I'm not sure about this statement.
await this.datablocksService.findOne({ where: { _id: id }, }); we have been implementing query like this in many places, I don't see what's the cons here..

Does it make sense to have more than "frontend" and "backend"? I would make this an enum. Or are you thinking there might be use cases where other microservices store most of their configs on the scicat backend?

We don’t want to lock it to just “frontend” or “backend”.
The idea is to let operators store any JSON config they want, and the backend shouldnt restrict it. They decide what config files they want to use, not us. It could be frontend-config, published-data-schema.json, proposal-ui-config etc

I would suggest just id as we ill store the whole configuration in one document... or at least, that was the idea last time we brainstormed.

sbliven · 2025-11-26T15:46:10Z

src/runtime-config/dto/runtime-config.dto.ts

+  })
+  @IsOptional()
+  @IsString()
+  description?: string;


What's the use of this? Just internal documentation?

That's the idea.
If a site admin uses this subsystem to store additional configurations, we should also have a name and a description for management purposes.

sbliven · 2025-11-26T15:50:03Z

src/runtime-config/utils.ts

@@ -0,0 +1,42 @@
+export const isPlainObject = (


Use common.utils.IsRecord()

sbliven · 2025-11-26T16:31:44Z

src/runtime-config/runtime-config.controller.ts

+  constructor(private readonly runtimeConfigService: RuntimeConfigService) {}
+
+  @AllowAny()
+  @Get("data/:id")


data strikes me as a surprising choice for the endpoint path. I see a close link between this functionality and the /admin/config endpoint. Maybe it could just be GET/PUT/PATCH /admin/config/:id? And there should be a way to get either the merged configuration or just the overridden values.

I do not remember why we decided to move it away from admin/config.
If I remember correctly, the reason for it is that this subsystem can become a store for additional blobs besides configurations. In our mind admin/config relates only to admin users, which is true for updating but not for retrieving.
Hopefully it makes sense, but I will be happy to discuss further

Junjiequan · 2025-11-27T10:08:45Z

I think we should discuss the core premise of this POC, which is that the configuration should be stored in the database instead of in files. While it would be convenient to be able to change the configuration on the fly, I'm worried that it will become even harder to track configuration changes. If we do it, I think there should be a clear process to "burn in" the configuration: export non-default settings to an env file, empty the database, and restart the backend with a clean configuration. Otherwise it's hard to keep track of changes.

We defintelly needs to discuss more about this, I just wanted to pull the idea into code, so we can have something to disccuss upon.

If i understand your proposal correctly, you are suggesting that:

the backend always starts with a empty config collection
it then loads the config files from disk (mounted or default)
creates a fresh config record based on provided files

So, the config file is the source of truth, config record is temporary copy for real time editing. once the backend restarts the temporary state resets and rebuilt from the file again.

Is it correct intepretation?

Can you add a description of the usage to the README or docs? Even if it changes later, it would be helpful to read how the merging works. I think you said that the database is considered canon and overrides anything in the config files or the defaults.

Sure, I will provide it as soon as possible.

How does this fit with /admin/config? Would this provide the merged "frontend" data?

It's up for discussion that wheter if we still want to keep /admin/config after we have the runtimeConfig endpoint.

Likewise, will calls to the configService be merged to the "backend" data? If so, we should double-check that there aren't any remaining direct calls to configuration(), as those would bypass the configService.

The backend data ( environment variables I assume ) is completly seperated logic and should not be relevant to the run time config service at all, they are out of scope this time at least. The run time service serves for frontend related configs only.

I guess you would add more endpoints

get the full, merged configuration
get/set individual properties (or would this just be done with PATCH application/json+merge?)
way to delete configuration changes (resetting from the files or defaults)

I imagine the entry sturcture will look like this:

[
  {
    "_id": "frontendConfig",
    "description": "Some text to be rendered on the UI",
    "data": {
      "...": "raw JSON blob"
    }
  },
  {
    "_id": "frontendTheme",
    "data": {
      "...": "raw JSON blob"
    }
  },
  {
    "_id": "someOtherConfig",
    "data": {
      "...": "raw JSON blob"
    }
  }
]

On the frontend we use jsonforms, which always submits the full JSON object. So, I’m not sure individual property endpoints add much value right now. JSONForms doesn’t send partial updates, so we’d essentially be duplicating logic.
For now, setting/getting the whole config blob seems enough, unless we decide to go with other json editors.

nitrosx · 2025-11-27T13:52:27Z

src/runtime-config/dto/runtime-config.dto.ts

I would also the fields createdBy, updatedAt, createdAt.

nitrosx · 2025-11-27T13:57:27Z

src/runtime-config/runtime-config.controller.ts

+  constructor(private readonly runtimeConfigService: RuntimeConfigService) {}
+
+  @AllowAny()
+  @Get("data/:id")


I do not remember why we decided to move it away from admin/config.
If I remember correctly, the reason for it is that this subsystem can become a store for additional blobs besides configurations. In our mind admin/config relates only to admin users, which is true for updating but not for retrieving.
Hopefully it makes sense, but I will be happy to discuss further

nitrosx · 2025-11-27T13:58:08Z

src/runtime-config/runtime-config.module.ts

The naming is perfect if we are going to use this subsystem to store only configurations.

nitrosx · 2025-11-27T13:58:24Z

src/runtime-config/runtime-config.service.ts

Same as my previous comment

nitrosx · 2025-11-27T13:59:14Z

src/runtime-config/utils.ts

@@ -0,0 +1,42 @@
+export const isPlainObject = (


nitrosx · 2025-11-27T14:04:49Z

@sbliven valid feedback!!!
Let me and Jay brainstorm once or twice more and get back to you.
We can discuss afterwards.

Junjiequan requested a review from a team as a code owner November 17, 2025 15:36

Junjiequan marked this pull request as draft November 17, 2025 15:36

Junjiequan added the ESS label Nov 18, 2025

Junjiequan changed the title ~~Swap 5086 scicat be poc store frontend config into the data~~ feat: add run time config management service Nov 18, 2025

Junjiequan marked this pull request as ready for review November 18, 2025 16:18

Junjiequan force-pushed the SWAP-5086-scicat-be-poc-store-frontend-config-into-the-data branch from e8b20cd to 26bb728 Compare November 18, 2025 16:18

Junjiequan self-assigned this Nov 18, 2025

Junjiequan added 4 commits November 21, 2025 13:21

PR

b4cb84a

init-2

1962f38

add history plugin for runtime service and reuse computeDeltaWithOrig…

105e4b3

…inals for sync change logging

revert frontend config

e7e05c5

Junjiequan force-pushed the SWAP-5086-scicat-be-poc-store-frontend-config-into-the-data branch from 6a92f73 to e7e05c5 Compare November 21, 2025 12:21

fix unit test

7d24f30

Junjiequan changed the title ~~feat: add run time config management service~~ feat(POC): add run time config management service Nov 25, 2025

bpedersen2 reviewed Nov 25, 2025

View reviewed changes

sbliven requested changes Nov 26, 2025

View reviewed changes

nitrosx requested changes Nov 27, 2025

View reviewed changes

feat(POC): add run time config management service #2355

Are you sure you want to change the base?

feat(POC): add run time config management service #2355

Uh oh!

Conversation

Junjiequan commented Nov 17, 2025

Description

Motivation

Fixes

Changes:

Tests included

Documentation

official documentation info

Uh oh!

sbliven commented Nov 25, 2025

Uh oh!

bpedersen2 left a comment

Choose a reason for hiding this comment

Uh oh!

sbliven left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Junjiequan Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nitrosx Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Junjiequan commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nitrosx commented Nov 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Junjiequan Nov 26, 2025 •

edited

Loading

nitrosx Nov 27, 2025 •

edited

Loading

Junjiequan commented Nov 27, 2025 •

edited

Loading