Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] - Upgrading nebari to 2025.2.1rc1 leads to broken conda-store due to remaints of Docker builds #2951

Open
viniciusdc opened this issue Feb 10, 2025 · 0 comments
Labels
needs: triage 🚦 Someone needs to have a look at this issue and triage type: bug 🐛 Something isn't working

Comments

@viniciusdc
Copy link
Contributor

viniciusdc commented Feb 10, 2025

Describe the bug

Another issue, in the latest conda-store migration, in [2025.1.1](https://github.com/conda-incubator/conda-store/releases/tag/2025.1.1) the DOCKER_BLOB column schema for the docker builds was removed, I initially assumed alambic would take care of any post DB update, but was wrong.

Image

In case any entries are present in the DB, the whole conda-store server breaks as seen below:

    |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    |     raw_response = await run_endpoint_function(
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    |     return await dependant.call(**values)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/conda_store_server/_internal/server/views/api.py", line 724, in api_list_environments_v1
    |     return paginated_api_response(
    |            ^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/conda_store_server/_internal/server/views/api.py", line 100, in paginated_api_response
    |     object_schema.model_validate(_).model_dump(exclude=exclude)
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/pydantic/main.py", line 627, in model_validate
    |     return cls.__pydantic_validator__.validate_python(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | pydantic_core._pydantic_core.ValidationError: 1 validation error for Environment
    | current_build.build_artifacts
    |   Error extracting attribute: LookupError: 'DOCKER_BLOB' is not among the defined enum values. Enum name: buildartifacttype. Possible values: DIRECTORY, LOCKFILE, LOGS, ..., _ [type=get_attribute_error, input_value=<unprintable Build object>, input_type=Build]
    |     For further information visit https://errors.pydantic.dev/2.10/v/get_attribute_error
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in __call__
    with recv_stream, send_stream, collapse_excgroups():
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/conda_store_server/_internal/server/app.py", line 257, in conda_store_middleware
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 154, in call_next
    raise app_exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/sessions.py", line 85, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/conda_store_server/_internal/server/views/api.py", line 724, in api_list_environments_v1
    return paginated_api_response(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/conda_store_server/_internal/server/views/api.py", line 100, in paginated_api_response
    object_schema.model_validate(_).model_dump(exclude=exclude)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/pydantic/main.py", line 627, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Environment
current_build.build_artifacts
  Error extracting attribute: LookupError: 'DOCKER_BLOB' is not among the defined enum values. Enum name: buildartifacttype. Possible values: DIRECTORY, LOCKFILE, LOGS, ..., _ [type=get_attribute_error, input_value=<unprintable Build object>, input_type=Build]
    For further information visit https://errors.pydantic.dev/2.10/v/get_attribute_error

Fortunately removing all affected builds in the DB through the API solves the issue, I am planning on encapsulating this as an automated step in the upgrade command that will perform the following:

  • Identify affected packages/builds (whose build_artifact is DOCKER_BLOB)
  • Preview the removal as a note to the user, and present with the option to save locally all builds' lock-files in case the user is interested in keeping it.

Expected behavior

fully functional conda-store server execution

OS and architecture in which you are running Nebari

Linux

How to Reproduce the problem?

Using any nebari deployment prior to 2025.2.1rc1 build a conda-store environment that requires a docker registry, then update nebari to its RC

Command output

Versions and dependencies used.

No response

Compute environment

GCP

Integrations

conda-store

Anything else?

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs: triage 🚦 Someone needs to have a look at this issue and triage type: bug 🐛 Something isn't working
Projects
Status: New 🚦
Development

No branches or pull requests

1 participant