PR Diff View takes a very long time to load: failed to run attr-check: context deadline exceeded #31600
Comments
|
Hi, we're experiencing the same issue, also after migrating to 1.22.1. We're also using a custom docker image, based on the official one, with a mysql database |
|
Same error here! |
|
@juanpablo-santos are you able to post your customizations? |
|
Hi, on the Dockerfile side, we copy some certificates, some gitea customizations (labels, license, hooks, README) and we set up GPG, asciidoctor and python, so we can render asciidoc files, upyter notebooks and restructuredtext. So, nothing fancy. Our |
|
Hi, we've noticed that having a high amount of branches might be involved on the long time to load. In our case one of the "usual" repos that suffers from this issue contains:
Proceeded to delete oldest merged / closed branches, so the repo is up to 73 branches now, we've noticed an increase in performance; it's still slow, but now is manageable HTH, |
lunny
added
the
performance/bigrepo
|
Hi, we've noticed that when we hit this issue, also going to the PR's commits also results in waiting a long time before they show up, so may be the diff view takes to long b/c of waiting a lot on retrieving the commits info? Some times all the commits of the PR take a long time, some times some commits involved in the PR take a long time to load. When loading those commits, at the Stacktrace Page of Gitea, I see that Gitea stuck on these commands: we've updated to 1.22.2 to see if #31754 would help, but long times still persist |
|
#31754 just delays the cat file commands invocation to avoid unnecessary usage of that command. |
|
Hi, issue doesn't seem related to big repos, we've just experienced the same issue on a PR of a repo with:
The PR experiencing this issue had 6 commits, 3 of them taking a really long time to render. Oh, and one last note, the files' rendering seems to always take around six minutes? |
|
One more update, as the six minutes waiting seem to be the git default timeout, we've set after that, we wait, hit one minute, and the diff of the PR/commit appears. Not sure anything is missing, everything seems in place :-? |
|
@juanpablo-santos thanks for the hint. We’ve been encountering the same error with some PRs/diffs since version 1.22. Reducing the git.timeout seems to be a workaround. |
jpraet
added
issue/regression
|
As a workaround, #31896 (comment) |
|
We encountered a timeout, and I downloaded the diagnosis report. |
|
Does anyone have any insights on what might be causing these git commands to hang or timeout (regression introduced between 1.21 and 1.22)?
In my instance, it mostly seems to happens for quite large PRs, but occasionally also for smaller PRs. Furthermore, it is an intermittent problem: it might hang and timeout after 90 seconds, but when you retry some time later you might get a correct response after 1 second. Also wondering about @juanpablo-santos comment:
Indeed: if I were to configure a smaller value for git.timeout of let's say 20 seconds as a workaround, what would be the potential risks? Are these commands performing any crucial functionality? |
FYI: Clarify the git command Stdin hanging problem (#26967) |
|
I made a simple test ( (the sample code is also in #33901) func TestAttributeReader(t *testing.T) {
ctx := t.Context()
timeout := 1 * time.Second // TODO: change it to a larger duration if needed
repoPath := filepath.Join(testReposDir, "language_stats_repo") // TODO: change it to your repo
commitRef := "HEAD" // TODO: change it to your repo
oneRound := func(t *testing.T, roundIdx int) {
ctx, cancel := context.WithTimeout(ctx, timeout)
_ = cancel
gitRepo, err := OpenRepository(ctx, repoPath)
require.NoError(t, err)
defer gitRepo.Close()
commit, err := gitRepo.GetCommit(commitRef)
require.NoError(t, err)
files, err := gitRepo.LsFiles()
require.NoError(t, err)
randomFiles := append(files, "any-file-1", "any-file-2")
t.Logf("Round %v with %d files", roundIdx, len(randomFiles))
attrReader, deferrable := gitRepo.CheckAttributeReader(commit.ID.String())
defer deferrable()
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
for {
file := randomFiles[mathRand.IntN(len(randomFiles))]
_, err := attrReader.CheckPath(file)
if err != nil {
for i := 0; i < 10; i++ {
_, _ = attrReader.CheckPath(file)
}
break
}
}
wg.Done()
}()
wg.Wait()
}
for i := 0; i < 100; i++ {
oneRound(t, i)
}
} |
|
According to your stacktrace, the hang-point is here, in the It is called by |
|
Two PRs: |
wxiaoguang
changed the title
|
Issue #34016 may resolve a related problem. |
|
I can now see these types of error logs after upgrading to 1.23.6:
|
|
Sorry but that's the content of |
Ok, I updated my comment because it's not really business-sensitive information after all. |
|
Let me try to summarize the current state of 1.23.6:
Am I right? |
|
Definitely 1. is right. Since the upgrade to 1.23.6 the page no longer times out as they did previously (after 90 seconds due to HA proxy timeout), which is already a great improvement. I do wonder: are we missing out any important information on the PR diff page when these slow attr-check have been aborted? About the frequency of |
outdated
And then check whether the timeout whether still occurs. -> Try to fix check-attr bug #34029 diff --git a/modules/git/repo_attribute.go b/modules/git/repo_attribute.go
index 89101e5af3..8a5e5fa983 100644
--- a/modules/git/repo_attribute.go
+++ b/modules/git/repo_attribute.go
@@ -280,7 +280,7 @@ func (wr *nulSeparatedAttributeWriter) Write(p []byte) (n int, err error) {
}
}
wr.tmp = append(wr.tmp, p...)
- return len(p), nil
+ return l, nil
}
func (wr *nulSeparatedAttributeWriter) ReadAttribute() <-chan attributeTriple { |
|
1.23 nightly is ready, it is a stable release and will be next 1.23.7:
I have managed to reproduce the problem and a fix is in 1.23-nightly now. Feel free to update and try. |
wxiaoguang
added
the
issue/needs-feedback
|
I upgraded to 1.23-nightly and it's looking good so far. Thanks @wxiaoguang! |
Description
Hello,
Since the update from 1.21.11 to 1.22.1 I have a problem displaying the differences of commits/PRs.
It takes several minutes for the pages to appear at all.
On the Stacktrace Page of Gitea i see that Gitea stuck on these commands:
/usr/bin/git cat-file --batch-check [repo_path: /data/git/repositories/test/test.git] (modules/gitrepo/gitrepo.go:31)/usr/bin/git cat-file --batch [repo_path: /data/git/repositories/test/test.git] (modules/gitrepo/gitrepo.go:31)git(dir:/data/git/repositories/test/test.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= check-attr --stdin -z --cached linguist-vendored linguist-generated linguist-documentation linguist-detectable linguist-language gitlab-languageAll these Git processes are in sleep state according to the system.
The Full log of Gitea is in the Gist. The Git check-attr Command start on line 17421 and Ends on Line 53584 with a Error :
failed to run attr-check. Error: context deadline exceededI cannot reproduce this issue in other Repos. The Issue is only in some PRs of this Repo (Mostly the ones with many files (>50)). The Repo has more than 9909 Commit with currently 280 Branches with a Size of 360 MiB (Git 152 MiB, LFS: 208 MiB).
Other Fact: When i run
straceon the Gitea Process with all Childs, then this Issue does not occur.Gitea Version
1.22.1
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
https://gist.github.com/andre-bergemann/580e08a7a4fdb44b67b3a1664375c6d7
Screenshots
No response
Git Version
2.45.2
Operating System
Ubuntu
How are you running Gitea?
Custom Container (For Gitea Config Changes) of the Gitea Docker Image (
gitea/gitea:1.22.1) on Ubuntu 22.04.2 VM with Docker Engine 23.0.1Database
PostgreSQL
The text was updated successfully, but these errors were encountered: