IFC-1436 Make work pools configurable #6346
Conversation
github-actions
bot
added
the
group/backend
CodSpeed Performance ReportMerging #6346 will not alter performanceComparing Summary
|
| default=[ | ||
| WorkerPoolDefinition( | ||
| name="infrahub-worker", | ||
| workflow_type=WorkflowType.INTERNAL | WorkflowType.CORE | WorkflowType.USER, |
There was a problem hiding this comment.
How would the configuration looks like when using environment variables if we want to configure multiple work_pools ?
There was a problem hiding this comment.
Also, not sure if having that as part of the configuration will work well .. ideally the worker pools should be pre-defined by the community and the enterprise version
There was a problem hiding this comment.
IIRC Pydantic is able to figure out these complex settings given a valid string representation of them. So in this case, it would be:
INFRAHUB_WORKFLOW_WORK_POOLS='[
{
"name": "infrahub-worker",
"workflow_type": 7,
"worker_type": "INFRAHUB_ASYNC",
"description": "Default Pool for internal tasks"
}
]'
The workflow type is more abstract as it's a int flag, but maybe this could be worked around by a validator for WorkerPoolDefinition.workflow_type.
The idea of making this a setting was it should be easy to change so we could have different values for community and enterprise without involving actual code change.
This will allow to create and use more work pools than the only one defined so far. Pools now define the type of workflows they are able to process. There can be more than one workflow type per pool, they can be combined using the `or` logical operator. If a workflow type is set in multiple pools, the last pool referencing will take precedence over the other ones. This means that only one work pool can handle a given type of workflows.
gmazoyer
force-pushed
the
gma-20250422-ifc1436
branch
from
de26125 to
6d1593d
Compare
This will allow to create and use more work pools than the only one defined so far. Pools now define the type of workflows they are able to process.
There can be more than one workflow type per pool, they can be combined using the
orlogical operator.If a workflow type is set in multiple pools, the last pool referencing will take precedence over the other ones. This means that only one work pool can handle a given type of workflows.