Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The lock resolve is slow for p-dml & the progress log of range task may be omitted #1577

Closed
you06 opened this issue Feb 14, 2025 · 0 comments · Fixed by #1584
Closed

The lock resolve is slow for p-dml & the progress log of range task may be omitted #1577

you06 opened this issue Feb 14, 2025 · 0 comments · Fixed by #1584

Comments

@you06
Copy link
Contributor

you06 commented Feb 14, 2025

Phenomenon

From the metrics, the resolve-lock stage of p-dml is very slow.

Image

And the logs does not log the progress every 30 seconds as expected, the range task runner only print the following log.

[2025/02/18 02:25:07.305 +00:00] [INFO] [range_task.go:167] ["range task started"] [name=pipelined-dml-commit-456089876295581703] [startKey=74800000000000007e5f69800000000000000103800000000000000103800000000185fa98] [endKey=74800000000000007e5f728000000005f5e100] [concurrency=8]
[2025/02/18 03:09:28.607 +00:00] [INFO] [range_task.go:276] ["range task finished"] [name=pipelined-dml-commit-456089876295581703] [startKey=74800000000000007e5f69800000000000000103800000000000000103800000000185fa98] [endKey=74800000000000007e5f728000000005f5e100] ["cost time"=44m21.302468224s] ["completed regions"=201]

Analysis

The progress log is print here:

select {
case <-statLogTicker.C:
logutil.Logger(ctx).Info("range task in progress",
zap.String("name", s.identifier),
zap.String("startKey", kv.StrKey(startKey)),
zap.String("endKey", kv.StrKey(endKey)),
zap.Int("concurrency", s.concurrency),
zap.Duration("cost time", time.Since(startTime)),
zap.Int("completed regions", s.CompletedRegions()))
default:
}

However, the progress log may not be printed if there are not such a lot of tasks, and it's waiting here.

@you06 you06 changed the title The progress log of range task may be omitted The lock resolve is slow for p-dml & the progress log of range task may be omitted Feb 18, 2025
@ti-chi-bot ti-chi-bot bot closed this as completed in 075b19f Feb 19, 2025
you06 added a commit to you06/client-go that referenced this issue Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant