Skip to content

dynamic-scheduler validation error reading from Redis DB table #7991

@matusdrobuliak66

Description

@matusdrobuliak66
  • dynamic-scheduler validation error reading from Redis DB table dynamic_services --> This is prohibiting to start any other new dynamic service.
log_level=ERROR | log_timestamp=2025-06-26 11:01:55,706 | log_source=servicelib.background_task:log_catch(388) | log_uid=None | log_oec=None| log_msg=Unhandled exception:
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/logging_utils.py", line 383, in log_catch
    yield
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/background_task.py", line 88, in _wrapper
    await func(*args, **kwargs)
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/background_task_utils.py", line 55, in _wrapper
    return await coro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_scheduler/services/status_monitor/_monitor.py", line 143, in _periodic_check_services_require_status_update
    await self._worker_check_services_require_status_update()
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_scheduler/services/status_monitor/_monitor.py", line 84, in _worker_check_services_require_status_update
    ] = await service_tracker.get_all_tracked_services(self.app)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_scheduler/services/service_tracker/_api.py", line 242, in get_all_tracked_services
    return await get_tracker(app).all()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_scheduler/services/service_tracker/_tracker.py", line 40, in all
    return {
           ^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_scheduler/services/service_tracker/_tracker.py", line 41, in <dictcomp>
    NodeID(k.decode().lstrip(_KEY_PREFIX)): TrackedServiceModel.from_bytes(v)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_dynamic_scheduler/services/service_tracker/_models.py", line 142, in from_bytes
    return cls(**unpacked_data)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/scu/.venv/lib/python3.11/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for TrackedServiceModel
dynamic_service_start.product_api_base_url
  Field required [type=missing, input_value={'key': 'simcore/services...ser_agent': 'undefined'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/missing

Happened on sim4life.io and tip.science --> Deleting data from the table in Redis helped.

Metadata

Metadata

Assignees

Labels

High Prioritya totally crucial bug/feature to be fixed asapa:dynamic-schedulerbugbuggy, it does not work as expected

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions