Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kwok is slow enabling deployments #1248

Closed
2 of 5 tasks
himred opened this issue Oct 24, 2024 · 1 comment
Closed
2 of 5 tasks

kwok is slow enabling deployments #1248

himred opened this issue Oct 24, 2024 · 1 comment
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@himred
Copy link

himred commented Oct 24, 2024

How to use it?

  • kwok
  • kwokctl --runtime=docker (default runtime)
  • kwokctl --runtime=binary
  • kwokctl --runtime=nerdctl
  • kwokctl --runtime=kind

What happened?

I'm using kwok with docker (kwok version v0.6.0 go1.22.3)
When I create 600 deployments (1 replicas, 1 namespace, 1 service for each deployment)
The 600 pods are created instantly
The 600 services are created instantly
The 600 deployments are created instantly BUT they are not ready: ie
ns99 fake-deploy-492 0/1 1 1 1m31s

On average on my beefy server it takes around 2 mins for them to be all ready, monitoring further shown that on average kwok validate 5 deployments/second.

I did the same with 50K deployments, it took almost 3 hours.
Any option to speed up this as there is no reason to take 0.2s to enable a deployment.

Any help, advices appreciated.

Thanks !

What did you expect to happen?

I expected to be fast.

How can we reproduce it (as minimally and precisely as possible)?

Yes, just deploy 600 simple deployments

Anything else we need to know?

No response

Kwok version

$ kwok --version
kwok version v0.6.0 go1.22.3 (linux/amd64)
$ kwokctl --version
kwokctl version v0.6.0 go1.22.3 (linux/amd64)

OS version

```console # On Linux: $ cat /etc/os-release PRETTY_NAME="Ubuntu 24.04.1 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.1 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble LOGO=ubuntu-logo $ uname -a inux Salimus 6.8.0-47-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 21:40:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
</details>
@himred himred added the kind/bug Categorizes issue or PR as related to a bug. label Oct 24, 2024
@himred himred closed this as completed Oct 24, 2024
@wzshiming
Copy link
Member

The bottleneck for this is the kube-scheduler; you need to increase its qps limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants