Skip to content

fix: set nginx worker_processes to fixed value to prevent OOMKill #1241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pranithraoibm wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: set nginx worker_processes to fixed value to prevent OOMKill #1241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ai-services/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
REGISTRY?=icr.io/ai-services-private
IMAGE=ai-services
TAG?=v0.0.228
TAG?=v0.0.229
CONTAINER_BUILDER?=podman
CREDS_ARG := $(if $(and $(REGISTRY_USER),$(REGISTRY_PASSWORD)),--creds="$(REGISTRY_USER):$(REGISTRY_PASSWORD)")

Expand Down
4 changes: 2 additions & 2 deletions ai-services/assets/applications/rag-cpu/podman/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ ui:
# @description Host port for the RAG UI. If unspecified, a random available port is assigned. Specify a port number to use a custom value.
port: ""
# @hidden
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.49
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.50

backend:
# @description Host port for the OpenAI-compatible RAG service. Defaults to unexposed; assign a port to enable external access.
Expand All @@ -28,7 +28,7 @@ digitizeUi:
# @description Host port for the DIGITIZE UI. If unspecified, a random available port is assigned. Specify a port number to use a custom value.
port: ""
# @hidden
image: icr.io/ai-services-cicd/digitize-ui:v0.0.30
image: icr.io/ai-services-cicd/digitize-ui:v0.0.31

ingest:
# @hidden
Expand Down
4 changes: 2 additions & 2 deletions ai-services/assets/applications/rag-dev/openshift/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
ui:
# @hidden
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.49
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.50

backend:
# @hidden
Expand Down Expand Up @@ -34,7 +34,7 @@ digitize:

digitizeUi:
# @hidden
image: icr.io/ai-services-cicd/digitize-ui:v0.0.30
image: icr.io/ai-services-cicd/digitize-ui:v0.0.31

instruct:
# @description API key for vLLM instruct service authentication. If empty, authentication is disabled. Provide a key to enable authentication.
Expand Down
4 changes: 2 additions & 2 deletions ai-services/assets/applications/rag-dev/podman/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ ui:
# @description Host port for the RAG UI. If unspecified, a random available port is assigned. Specify a port number to use a custom value.
port: ""
# @hidden
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.49
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.50

backend:
# @description Host port for the OpenAI-compatible RAG service. Defaults to unexposed; assign a port to enable external access.
Expand All @@ -28,7 +28,7 @@ digitizeUi:
# @description Host port for the DIGITIZE UI. If unspecified, a random available port is assigned. Specify a port number to use a custom value.
port: ""
# @hidden
image: icr.io/ai-services-cicd/digitize-ui:v0.0.30
image: icr.io/ai-services-cicd/digitize-ui:v0.0.31

ingest:
# @hidden
Expand Down
4 changes: 2 additions & 2 deletions ai-services/assets/applications/rag/openshift/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
ui:
# @hidden
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.49
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.50

backend:
# @hidden
Expand Down Expand Up @@ -34,7 +34,7 @@ digitize:

digitizeUi:
# @hidden
image: icr.io/ai-services-cicd/digitize-ui:v0.0.30
image: icr.io/ai-services-cicd/digitize-ui:v0.0.31

instruct:
# @hidden
Expand Down
4 changes: 2 additions & 2 deletions ai-services/assets/applications/rag/podman/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ ui:
# @description Host port for the RAG UI. If unspecified, a random available port is assigned. Specify a port number to use a custom value.
port: ""
# @hidden
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.49
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.50

backend:
# @description Host port for the OpenAI-compatible RAG service. Defaults to unexposed; assign a port to enable external access.
Expand All @@ -28,7 +28,7 @@ digitizeUi:
# @description Host port for the DIGITIZE UI. If unspecified, a random available port is assigned. Specify a port number to use a custom value.
port: ""
# @hidden
image: icr.io/ai-services-cicd/digitize-ui:v0.0.30
image: icr.io/ai-services-cicd/digitize-ui:v0.0.31

ingest:
# @hidden
Expand Down
4 changes: 2 additions & 2 deletions ai-services/assets/catalog/openshift/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
ui:
image: icr.io/ai-services-cicd/catalog-ui:v0.0.49
image: icr.io/ai-services-cicd/catalog-ui:v0.0.50
resources:
requests:
memory: "512Mi"
Expand All @@ -9,7 +9,7 @@ ui:
cpu: "500m"

backend:
image: icr.io/ai-services-cicd/ai-services:v0.0.228
image: icr.io/ai-services-cicd/ai-services:v0.0.229
runtime: "openshift"
adminPasswordHash: ""
resources:
Expand Down
4 changes: 2 additions & 2 deletions ai-services/assets/catalog/podman/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
ui:
port: ""
image: icr.io/ai-services-cicd/catalog-ui:v0.0.49
image: icr.io/ai-services-cicd/catalog-ui:v0.0.50

backend:
port: ""
image: icr.io/ai-services-cicd/ai-services:v0.0.228
image: icr.io/ai-services-cicd/ai-services:v0.0.229
runtime: ""
adminPasswordHash: ""
podman:
Expand Down
2 changes: 1 addition & 1 deletion ai-services/assets/services/chat/openshift/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
ui:
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.49
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.50
resources:
requests:
memory: "512Mi"
Expand Down
2 changes: 1 addition & 1 deletion ai-services/assets/services/chat/podman/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
ui:
port: ""
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.49
image: icr.io/ai-services-cicd/chatbot-ui:v0.0.50

backend:
port: ""
Expand Down
2 changes: 1 addition & 1 deletion ai-services/assets/services/digitize/openshift/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ digitize:
memory: "50Gi"

digitizeUi:
image: icr.io/ai-services-cicd/digitize-ui:v0.0.30
image: icr.io/ai-services-cicd/digitize-ui:v0.0.31
resources:
requests:
cpu: "50m"
Expand Down
2 changes: 1 addition & 1 deletion ai-services/assets/services/digitize/podman/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ digitize:
database: "digitize_metadata"

digitizeUi:
image: icr.io/ai-services-cicd/digitize-ui:v0.0.30
image: icr.io/ai-services-cicd/digitize-ui:v0.0.31

postgres:
image: icr.io/ai-services-cicd/postgres:18-4
Expand Down
2 changes: 1 addition & 1 deletion ui/catalog/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
REGISTRY ?= icr.io/ai-services-private
IMAGE ?= catalog-ui
TAG ?= v0.0.49
TAG ?= v0.0.50
CREDS_ARG := $(if $(and $(REGISTRY_USER),$(REGISTRY_PASSWORD)),--creds="$(REGISTRY_USER):$(REGISTRY_PASSWORD)")

CONTAINER_BUILDER ?= podman
Expand Down
4 changes: 2 additions & 2 deletions ui/catalog/nginx.conf.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
worker_processes auto;
worker_processes 2;

error_log /var/log/nginx/error.log warn;
pid /tmp/nginx.pid;

events {
worker_connections 1024;
worker_connections 256;
}

http {
Expand Down
2 changes: 1 addition & 1 deletion ui/chatbot/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
REGISTRY?=icr.io/ai-services-private
IMAGE=chatbot-ui
TAG?=v0.0.49
TAG?=v0.0.50
CREDS_ARG := $(if $(and $(REGISTRY_USER),$(REGISTRY_PASSWORD)),--creds="$(REGISTRY_USER):$(REGISTRY_PASSWORD)")

CONTAINER_BUILDER?=podman
Expand Down
4 changes: 2 additions & 2 deletions ui/chatbot/nginx.conf.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
worker_processes auto;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /tmp/nginx.pid;


events {
worker_connections 1024;
worker_connections 256;
}


Expand Down
2 changes: 1 addition & 1 deletion ui/digitize/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
REGISTRY ?= icr.io/ai-services-private
IMAGE ?= digitize-ui
TAG ?= v0.0.30
TAG ?= v0.0.31
CREDS_ARG := $(if $(and $(REGISTRY_USER),$(REGISTRY_PASSWORD)),--creds="$(REGISTRY_USER):$(REGISTRY_PASSWORD)")

CONTAINER_BUILDER ?= podman
Expand Down
4 changes: 2 additions & 2 deletions ui/digitize/nginx.conf.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
worker_processes auto;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /tmp/nginx.pid;

events {
worker_connections 1024;
worker_connections 256;
}
Comment on lines +1 to 8

@mayuka-c mayuka-c Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did we come up with this number?
How exactly will this be impacted?. Did we do thorough validations and ensure no issues?

@mayuka-c mayuka-c Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc. @yussufsh @dharaneeshvrd @manju956

@dharaneeshvrd dharaneeshvrd Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pranithraoibm can you please explain what is this for?

@yussufsh yussufsh Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pranithraoibm please put a comment to change the numbers when someone changes the limits. Better add comment at both the places.

@pranithraoibm pranithraoibm Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this change was something that Bob had suggested

worker_connections defines the maximum number of simultaneous connections per worker process.
Total max connections = worker_processes × worker_connections
In our case, it is 512 for catalog and 256 for chatbot and digitize

256 was chosen because it is proportionate to what the pod can actually handle given its CPU limit (500m and 50m).
So this right-sizes the connection pool to match the worker count and CPU budget, reducing unnecessary memory reservation.

Just wanted to highlight this, we can revert back if this seems as a unnecessary improvement.


http {
Expand Down
Loading