-
Notifications
You must be signed in to change notification settings - Fork 24
Labels
Description
Description
Opening lumigator throw 500 server errors when accessing the GET /jobs endpoint.
I tried deleting all datasets and experiments with no luck, also tried make clean-all, but it seems like the jobs are partially persisted somewhere else somehow leading to this error
Reproduction
- Run
make clean-all - Run
make local-up - Open lumigator frontend
localhost:80orlocalhost:3000if running frontend dev server vianpm run dev
Relevant log output
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
backend-1 | result = await app( # type: ignore[func-returns-value]
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
backend-1 | return await self.app(scope, receive, send)
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
backend-1 | await super().__call__(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
backend-1 | await self.middleware_stack(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
backend-1 | raise exc
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
backend-1 | await self.app(scope, receive, _send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
backend-1 | await self.simple_response(scope, receive, send, request_headers=headers)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 144, in simple_response
backend-1 | await self.app(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
backend-1 | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend-1 | raise exc
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
backend-1 | await app(scope, receive, sender)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
backend-1 | await self.middleware_stack(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
backend-1 | await route.handle(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
backend-1 | await self.app(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
backend-1 | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend-1 | raise exc
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
backend-1 | await app(scope, receive, sender)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
backend-1 | response = await f(request)
backend-1 | ^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
backend-1 | raw_response = await run_endpoint_function(
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
backend-1 | return await run_in_threadpool(dependant.call, **values)
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
backend-1 | return await anyio.to_thread.run_sync(func)
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
backend-1 | return await get_async_backend().run_sync_in_worker_thread(
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
backend-1 | return await future
backend-1 | ^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 962, in run
backend-1 | result = context.run(func, *args)
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/backend/api/routes/jobs.py", line 125, in list_jobs
backend-1 | jobs = service.list_jobs(skip, limit, job_types)
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/backend/services/jobs.py", line 569, in list_jobs
backend-1 | items=[self.get_job(record.id) for record in records],
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/backend/services/jobs.py", line 569, in <listcomp>
backend-1 | items=[self.get_job(record.id) for record in records],
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/backend/services/jobs.py", line 543, in get_job
backend-1 | job_status = self.ray_client.get_job_status(job_id)
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/ray/dashboard/modules/job/sdk.py", line 424, in get_job_status
backend-1 | return self.get_job_info(job_id).status
backend-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/ray/dashboard/modules/job/sdk.py", line 359, in get_job_info
backend-1 | self._raise_error(r)
backend-1 | File "/mzai/lumigator/backend/.venv/lib/python3.11/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 283, in _raise_error
backend-1 | raise RuntimeError(
backend-1 | RuntimeError: Request failed with status code 404: Job 67036c9b-f261-43a8-8831-152ec514f67b does not exist.
backend-1 | INFO: 192.168.65.1:36931 - "GET /api/v1/datasets/ HTTP/1.1" 200 OKExpected behavior
Jobs endpoint should not fail
System Info
- main branch
- macOS
Have you searched for similar issues before submitting this one?
- Yes, I have searched for similar issues