Skip to content

Tuning resource requirements prod airbyte #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Tuning resource requirements prod airbyte #100

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1ec12f6
set sensible limits based on the grafana charts on the server and wor…
Ishankoradia Oct 23, 2025
2394644
remove this limit now
Ishankoradia Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 26 additions & 18 deletions dalgo-kubernetes/airbyte-kubernetes/prod/helm/values1.8.3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -523,13 +523,17 @@ server:
## cpu: 200m
## memory: 1Gi
# -- The resources limits for the server container
limits: {}
limits:
cpu: 1.5
memory: 3Gi
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
# -- The requested resources for the server container
requests: {}
requests:
cpu: 1
memory: 2Gi

# -- Node labels for pod assignment, see https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector:
Expand Down Expand Up @@ -740,13 +744,17 @@ worker:
## cpu: 200m
## memory: 1Gi
#! -- The resources limits for the worker container
limits: {}
limits:
cpu: 1.5
memory: 3Gi
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
# -- The requested resources for the worker container
requests: {}
requests:
cpu: 1
memory: 2Gi

# -- Node labels for pod assignment, see https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector:
Expand Down Expand Up @@ -810,14 +818,6 @@ worker:
secretKeyRef:
name: airbyte-config-secrets
key: S3_LOG_BUCKET
- name: MAX_SYNC_WORKERS
value: "4"
- name: MAX_CHECK_WORKERS
value: "5"
- name: MAX_SPEC_WORKERS
value: "5"
- name: MAX_DISCOVER_WORKERS
value: "5"
## Examples (when using `worker.containerSecurityContext.readOnlyRootFilesystem=true`):
## extraVolumeMounts:
## - name: tmpdir
Expand Down Expand Up @@ -851,7 +851,7 @@ worker:
workload-launcher:
enabled: true
# -- Number of workload launcher replicas
replicaCount: 1
replicaCount: 2

image:
# -- The repository to use for the airbyte workload launcher image.
Expand Down Expand Up @@ -923,13 +923,17 @@ workload-launcher:
## cpu: 200m
## memory: 1Gi
# -- The resources limits for the workload launcher container
limits: {}
limits:
cpu: 512m
memory: 1Gi
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
# -- The requested resources for the workload launcher container
requests: {}
requests:
cpu: 250m
memory: 512Mi

# -- Node labels for pod assignment, see https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector:
Expand Down Expand Up @@ -2167,7 +2171,7 @@ workload-api-server:
bearerToken: token

# -- workload-api-server replicas
replicaCount: 1
replicaCount: 2

image:
# -- The repository to use for the airbyte-workload-api-server image.
Expand Down Expand Up @@ -2233,13 +2237,17 @@ workload-api-server:
## cpu: 200m
## memory: 1Gi
# -- The resources limits for the airbyte-workload-api-server container
limits: {}
limits:
cpu: 500m
memory: 1Gi
## Examples:
## requests:
## memory: 256Mi
## cpu: 250m
# -- The requested resources for the airbyte-workload-api-server container
requests: {}
requests:
cpu: 250m
memory: 512Mi

log:
# -- The log level at which to log
Expand Down