Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Huge pull request creation timeout #27967

Closed
TitaniumHocker opened this issue Nov 8, 2023 · 9 comments
Closed

Huge pull request creation timeout #27967

TitaniumHocker opened this issue Nov 8, 2023 · 9 comments
Labels
performance/speed performance issues with slow downs type/bug

Comments

@TitaniumHocker
Copy link
Contributor

Description

I have huge pull request with more that 1kk diff lines and it always fails with timeout.
I raised the timeout on the reverse proxy side to 5 minutes, and then to 10 minutes, but this did not help. I don’t know if there is any point in raising the timeout anymore.

Here is some errors from logs:

Nov 08 17:04:46 gitea-hostname gitea[699]: 2023/11/08 17:04:46 ...ll/patch_unmerged.go:164:unmergedFiles() [E] Unable to run ls-files -u -z! Error: git ls-files -u -z: context deadline exceeded
Nov 08 17:04:46 gitea-hostname gitea[699]: 2023/11/08 17:04:46 ...ers/web/repo/pull.go:1297:CompareAndPullRequestPost() [E] NewPullRequest: signal: killed

I tried to add more RAM and CPU to VM, but it doesn't help. Judging by htop, this operation does not waste a lot of resources.

Gitea Version

1.20.5

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

gateway-timeout
gitea-timeout

Git Version

2.30.2

Operating System

Debian 11

How are you running Gitea?

Selfhosted with systemd unit, downloaded from releases page on github.
VM instance has 8GB of RAM and 4 vCPU.

Database

PostgreSQL

@KN4CK3R
Copy link
Member

KN4CK3R commented Nov 8, 2023

You may need to adjust these timeouts:

;; Git Operation timeout in seconds
;[git.timeout]
;DEFAULT = 360
;MIGRATE = 600
;MIRROR = 300
;CLONE = 300
;PULL = 300
;GC = 60

@TitaniumHocker
Copy link
Contributor Author

You may need to adjust these timeouts:

I don't see timeout for git ls-files in this list. Besides, my operation timeouts on reverse proxy side. I don't want to increase timeouts - 10 minutes alreary too much.

@KN4CK3R
Copy link
Member

KN4CK3R commented Nov 9, 2023

git ls-files uses the DEFAULT timeout setting.

@TitaniumHocker
Copy link
Contributor Author

TitaniumHocker commented Nov 9, 2023

git ls-files uses the DEFAULT timeout setting.

Seems like it doesn't work, process fails on reverse proxy timeout in 10 minutes, not in 6 minutes with default value.

@lschwahn
Copy link

Hi,

we are observing a similar behavior:

In our case it's not, that the pull request itself is that big. But the amount of files identified by following Function ist ~ 1000.

func readUnmergedLsFileLines(ctx context.Context, tmpBasePath string, outputChan chan *lsFileLine) {

Because of this the following loop runs several minutes, resulting in a timeout because of our Reverse-Proxy (which was set to
60 secs).

for line := range lsFileLineChan {

We set the reverse-proxy timeout to 6 Minutes, to match the gitea default timeout values. As a workaround this "resolved" the
Proxy-Timeout-Errors. But the time spent to create a pull request is still not satisfying.

Gitea Version
1.21.0

Git Version
2.40

Operating System
Rocky-Linux 8

How are you running Gitea?
Docker. We reproduced the behavior on three different machines with Rocky Linux 8 installed:

PowerEdge R630: CPU: 2 * Intel Xeon E5-2630 v3 (2.4GHz; 8 Cores), Storage: SAS, Memory: 128GB
PowerEdge R740: CPU: 2* Intel Xeon Gold 6126 (2.6GHz; 12 Cores), Storage: NVMe, Memory: 768GB
Virtual Server (Hyper-V): CPU: 4 Cores, Memory: 8GB

Database
MariaDB

@achepukov
Copy link

As a workaround you can create PR based on branch equals to base branch, and then push new commits into it.

@lunny lunny added the performance/speed performance issues with slow downs label Apr 4, 2024
@bartvdbraak
Copy link

We're also running into this issue, unfortunately.

Even with:

[git.timeout]
DEFAULT = 600

@brechtvl
Copy link
Contributor

It appears that running git update-index --remove for every individual file is slow. Batching them all together is much faster.

I have a change here that helps for file removal, but not file addition yet. I won't have time to finish this, but maybe someone else will pick it up.
https://github.com/blender/gitea/commits/fix-slow-check-many-files/

brechtvl added a commit to brechtvl/gitea that referenced this issue Jul 3, 2024
Running git update-index for every individual file is slow, so add and
remove everything with a single git command.

When such a commit lands in the default branch, it could cause PR creation
and patch checking for all open PRs to be slow, or time out entirely. For
example, a commit that removes 1383 files was measured to take more than
60 seconds and timed out. With this change checking takes about a second.

Related to go-gitea#27967, though this will not help with commits that change many
lines in few files.
techknowlogick pushed a commit that referenced this issue Jul 4, 2024
…31548)

Running git update-index for every individual file is slow, so add and
remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR
creation and patch checking for all open PRs to be slow, or time out
entirely. For example, a commit that removes 1383 files was measured to
take more than 60 seconds and timed out. With this change checking took
about a second.

This is related to #27967, though this will not help with commits that
change many lines in few files.
GiteaBot pushed a commit to GiteaBot/gitea that referenced this issue Jul 4, 2024
…o-gitea#31548)

Running git update-index for every individual file is slow, so add and
remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR
creation and patch checking for all open PRs to be slow, or time out
entirely. For example, a commit that removes 1383 files was measured to
take more than 60 seconds and timed out. With this change checking took
about a second.

This is related to go-gitea#27967, though this will not help with commits that
change many lines in few files.
silverwind pushed a commit that referenced this issue Jul 5, 2024
…31548) (#31560)

Backport #31548 by @brechtvl

Running git update-index for every individual file is slow, so add and
remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR
creation and patch checking for all open PRs to be slow, or time out
entirely. For example, a commit that removes 1383 files was measured to
take more than 60 seconds and timed out. With this change checking took
about a second.

This is related to #27967, though this will not help with commits that
change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to blender/gitea that referenced this issue Jul 15, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
bartvdbraak pushed a commit to bartvdbraak/gitea that referenced this issue Nov 21, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
blender-it pushed a commit to blender/gitea that referenced this issue Nov 22, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
blender-it pushed a commit to blender/gitea that referenced this issue Nov 22, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
blender-it pushed a commit to blender/gitea that referenced this issue Nov 28, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
blender-it pushed a commit to blender/gitea that referenced this issue Dec 4, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
blender-it pushed a commit to blender/gitea that referenced this issue Dec 4, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
blender-it pushed a commit to blender/gitea that referenced this issue Dec 4, 2024
… files

Running git update-index for every individual file is slow, so add and remove everything with a single git command.

When such a big commit lands in the default branch, it could cause PR creation and patch checking for all open PRs to be slow, or time out entirely. For example, a commit that removes 1383 files was measured to take more than 60 seconds and timed out. With this change checking took about a second.

This is related to go-gitea#27967, though this will not help with commits that change many lines in few files.

Co-authored-by: Brecht Van Lommel <[email protected]>
@lunny
Copy link
Member

lunny commented Dec 5, 2024

Duplicate of #31600

@lunny lunny marked this as a duplicate of #31600 Dec 5, 2024
@lunny lunny closed this as not planned Won't fix, can't repro, duplicate, stale Dec 5, 2024
@go-gitea go-gitea locked as resolved and limited conversation to collaborators Mar 6, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
performance/speed performance issues with slow downs type/bug
Projects
None yet
Development

No branches or pull requests

7 participants