Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] - Conda-Store empty UI listing for namespace on 2025.1.1rc1 #2927

Closed
viniciusdc opened this issue Jan 29, 2025 · 4 comments
Closed

[BUG] - Conda-Store empty UI listing for namespace on 2025.1.1rc1 #2927

viniciusdc opened this issue Jan 29, 2025 · 4 comments
Labels
area: integration/conda-store block-release ⛔️ Must be completed for release impact: high 🟥 This issue affects most of the nebari users or is a critical issue needs: discussion 💬 Needs discussion with the rest of the team type: bug 🐛 Something isn't working

Comments

@viniciusdc
Copy link
Contributor

Describe the bug

While testing the next release candidate, i.e 2025.1.1rc1 on a pre-deployed nebari, I noticed that intenrlay to the conda-store-server pod, an internal error was ocuround with regarding to the e validation of the Database ORM Namespace object, which leaded to a complete failed to load namespace list in the actual UI as seen bellow:

Image

The root issue has been forwarded to conda-store already under conda-incubator/conda-store#1062, and a fix will likely be included in an upcking conda-store release. Though, this leaves us with a way to handle this properly on nebari right now:

  • for systems that already tested out the newest nebari rc, removing the conda-store upgrade from this release could cause issues, since we had mentions of problems with the conda-store being rollback in the past -- mostly DB related
  • The current workaround involves a change in the DB Namespaces table to update any metadata_ nullable fields into empty dictionaries (more info in the linked issue above)

It seems that the best course of action can be summarized in two scenarios:

  • Hold the nebari release until the issue on conda-store is addressed upstream, and a new version is cut;
  • Include notes or an automated step under the upgrade command to automatic update the PostregresSQL DB associated with conda-store;

Expected behavior

Namespace should show up in conda-store UI and usual user experience should succeed;

OS and architecture in which you are running Nebari

Linux

How to Reproduce the problem?

Notes on reproducatbilty in the original conda-store issue

Command output

Versions and dependencies used.

Nebari 2025.1.1rc1

Compute environment

GCP

Integrations

conda-store

Anything else?

Open for discussions

@viniciusdc viniciusdc added needs: triage 🚦 Someone needs to have a look at this issue and triage type: bug 🐛 Something isn't working labels Jan 29, 2025
@viniciusdc viniciusdc added the needs: discussion 💬 Needs discussion with the rest of the team label Jan 29, 2025
@viniciusdc
Copy link
Contributor Author

my opinion on this would be to wait for an upstream fix if that waiting is reasonable or integrate both a note in the upgrade command to point the user to a DOCs FAQ with the workaround and instructions and a prompt for handling the automatically -- though this auto part might be a bit complex since we need to validate the current state of the DB before attempting any changes, then updating the table, while also keeping a port forward from the Postgres service to actually compute the queries

@viniciusdc viniciusdc added impact: high 🟥 This issue affects most of the nebari users or is a critical issue area: integration/conda-store block-release ⛔️ Must be completed for release and removed needs: triage 🚦 Someone needs to have a look at this issue and triage labels Jan 29, 2025
@viniciusdc viniciusdc mentioned this issue Jan 29, 2025
25 tasks
@viniciusdc
Copy link
Contributor Author

The recent release on conda-store might address the root cause of the issue; I asserted the workaround fixes the problem. It seems that only +2y old nebari deployments should be affected by this issue in particular since the latest namespaces were already adopting the correct default value for the metadata_ field -- I suspect this comes from versions that relate back to the Qhub+Conda-store era.

Nevertheless, while testing the new release of the conda-store, I found a new issue regarding the alembic migration and possibly a race condition for updating the DB from the main server and the celery workers. I am currently waiting on that.

@viniciusdc
Copy link
Contributor Author

All working with the latest changes from #2931

@viniciusdc
Copy link
Contributor Author

Fixed in upstream conda-store

@github-project-automation github-project-automation bot moved this from New 🚦 to Done 💪🏾 in 🪴 Nebari Project Management Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: integration/conda-store block-release ⛔️ Must be completed for release impact: high 🟥 This issue affects most of the nebari users or is a critical issue needs: discussion 💬 Needs discussion with the rest of the team type: bug 🐛 Something isn't working
Projects
Status: Done 💪🏾
Development

Successfully merging a pull request may close this issue.

1 participant