[BUG] - Conda-Store empty UI listing for namespace on 2025.1.1rc1

[viniciusdc]

Describe the bug

While testing the next release candidate, i.e 2025.1.1rc1 on a pre-deployed nebari, I noticed that intenrlay to the conda-store-server pod, an internal error was ocuround with regarding to the e validation of the Database ORM Namespace object, which leaded to a complete failed to load namespace list in the actual UI as seen bellow:

The root issue has been forwarded to conda-store already under conda-incubator/conda-store#1062, and a fix will likely be included in an upcking conda-store release. Though, this leaves us with a way to handle this properly on nebari right now:

• for systems that already tested out the newest nebari rc, removing the conda-store upgrade from this release could cause issues, since we had mentions of problems with the conda-store being rollback in the past -- mostly DB related
• The current workaround involves a change in the DB Namespaces table to update any metadata_ nullable fields into empty dictionaries (more info in the linked issue above)

It seems that the best course of action can be summarized in two scenarios:

• Hold the nebari release until the issue on conda-store is addressed upstream, and a new version is cut;
• Include notes or an automated step under the upgrade command to automatic update the PostregresSQL DB associated with conda-store;

Expected behavior

Namespace should show up in conda-store UI and usual user experience should succeed;

OS and architecture in which you are running Nebari

Linux

How to Reproduce the problem?

Notes on reproducatbilty in the original conda-store issue

Command output

Versions and dependencies used.

Nebari 2025.1.1rc1

Compute environment

GCP

Integrations

conda-store

Anything else?

Open for discussions

[viniciusdc]

my opinion on this would be to wait for an upstream fix if that waiting is reasonable or integrate both a note in the upgrade command to point the user to a DOCs FAQ with the workaround and instructions and a prompt for handling the automatically -- though this auto part might be a bit complex since we need to validate the current state of the DB before attempting any changes, then updating the table, while also keeping a port forward from the Postgres service to actually compute the queries

[viniciusdc]

The recent release on conda-store might address the root cause of the issue; I asserted the workaround fixes the problem. It seems that only +2y old nebari deployments should be affected by this issue in particular since the latest namespaces were already adopting the correct default value for the metadata_ field -- I suspect this comes from versions that relate back to the Qhub+Conda-store era.

Nevertheless, while testing the new release of the conda-store, I found a new issue regarding the alembic migration and possibly a race condition for updating the DB from the main server and the celery workers. I am currently waiting on that.

[viniciusdc]

All working with the latest changes from #2931

[viniciusdc]

Fixed in upstream conda-store