fix: set nginx worker_processes to fixed value to prevent OOMKill - #1241
fix: set nginx worker_processes to fixed value to prevent OOMKill#1241pranithraoibm wants to merge 1 commit into
Conversation
Signed-off-by: Pranith Rao <pranith.rao@ibm.com>
Shubhamag12
left a comment
Shubhamag12
left a comment
There was a problem hiding this comment.
lgtm
one question: are these configurations tested?
| worker_processes 1; | ||
|
|
||
| error_log /var/log/nginx/error.log warn; | ||
| pid /tmp/nginx.pid; | ||
|
|
||
| events { | ||
| worker_connections 1024; | ||
| worker_connections 256; | ||
| } |
There was a problem hiding this comment.
How did we come up with this number?
How exactly will this be impacted?. Did we do thorough validations and ensure no issues?
There was a problem hiding this comment.
There was a problem hiding this comment.
@pranithraoibm can you please explain what is this for?
There was a problem hiding this comment.
@pranithraoibm please put a comment to change the numbers when someone changes the limits. Better add comment at both the places.
There was a problem hiding this comment.
So this change was something that Bob had suggested
worker_connections defines the maximum number of simultaneous connections per worker process.
Total max connections = worker_processes × worker_connections
In our case, it is 512 for catalog and 256 for chatbot and digitize
256 was chosen because it is proportionate to what the pod can actually handle given its CPU limit (500m and 50m).
So this right-sizes the connection pool to match the worker count and CPU budget, reducing unnecessary memory reservation.
Just wanted to highlight this, we can revert back if this seems as a unnecessary improvement.
5 participants
nginx worker_processes auto reads /proc/cpuinfo (node CPU count) instead of the pod's cpu limit, causing pods to spawn 48 worker processes on a 48-vCPU node and get OOMKilled.
Set worker_processes to a fixed value aligned with each pod's cpu limit:
Also reduced worker_connections from 1024 to 256 to match the reduced worker footprint.