newlog: add filter, dedup and counter functions #4702
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds new features to reduce log volume in Newlog by implementing three log reduction mechanisms: filtering, deduplication, and counting.
- Introduces a deduplication feature for error logs using a ring buffer.
- Implements a log counter that appends occurrence counts to log entries.
- Adds configurable log filtering and updates global configuration settings accordingly.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/newlog/cmd/dedup.go | Adds deduplication functions for real‐time log reduction |
| pkg/newlog/cmd/counter.go | Implements a counter to tag and suppress duplicate log entries |
| pkg/newlog/cmd/dedup_test.go | Provides tests for deduplication logic |
| pkg/newlog/cmd/filter.go | Introduces log filtering using configurable filename filters |
| pkg/newlog/cmd/newlogd.go | Updates main log processing to integrate the new deduplication and filtering logic |
| pkg/pillar/types/global.go | Adds new global settings keys for deduplication, log counting, and filtering |
| pkg/newlog/cmd/newlogd_test.go | Minor test update for gzip log parsing |
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
c0462b9 to
e947722
Compare
There was a problem hiding this comment.
Pull Request Overview
This pull request adds three new log reduction mechanisms—filtering, deduplication, and counting—to help reduce excessive logs. Key changes include adding deduplication functionality with a sliding window (pkg/newlog/cmd/dedup.go), implementing log counting (pkg/newlog/cmd/counter.go), introducing log filtering (pkg/newlog/cmd/filter.go), and integrating these features via configuration into the log compression routine (pkg/newlog/cmd/newlogd.go).
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/newlog/cmd/dedup.go | New deduplication logic with a sliding window for error log entries |
| pkg/newlog/cmd/counter.go | New log counting functionality to annotate log entries |
| pkg/newlog/cmd/dedup_test.go | Test cases for deduplication with an identified iteration bug |
| pkg/newlog/cmd/filter.go | New log filtering mechanism based on filename criteria |
| pkg/pillar/types/global.go | Added configuration items for deduplication and filtering settings |
| pkg/newlog/cmd/newlogd.go | Integrated log deduplication, counting, and filtering into file compression |
| pkg/newlog/cmd/newlogd_test.go | Test scaffolding for log compression and integration testing |
europaul
force-pushed
the
log-filtering-and-dedup
branch
2 times, most recently
from
8572d7f to
a2357dc
Compare
|
Waiting on #4703 to be merged to import the newer version of pillar with the right global config parameters. |
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
a2357dc to
e2ff9b3
Compare
|
Yetus seems to be bailing out again because the PR includes too many updated vendor dependencies. |
|
@eriknordmark could you rebase this PR, go modules for pkg/newlog were updated by dependabot... |
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
e2ff9b3 to
aac93d6
Compare
@rene done |
There was a problem hiding this comment.
I see yetus/golangcilint is failing with the output
pkg/newlog/level=error msg="Running error: context loading failed: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: downloading go1.24 (linux/amd64)\ngo: download go1.24 for linux/amd64: toolchain not available\n"
pkg/newlog/level=warning msg="Failed to discover go env: failed to run 'go env': exit status 1"
@eriknordmark nvm, I found it in scan-results artefact |
|
@europaul , LGTM, just left a few comments.... |
pkg/newlog/cmd/dedup.go
Outdated
|
|
||
| // deduplicateLogs can be used to deduplicate logs on the fly reading from a channel | ||
| // and writing to another channel | ||
| func deduplicateLogs(in <-chan inputEntry, out chan<- inputEntry) { |
There was a problem hiding this comment.
Where is this actually used outside of tests?
There was a problem hiding this comment.
my first attempt was to dedup logs on the fly as they come into the newlogd - then this function was used. I'm still not sure which is the better way: this or deduplicating while compressing - that's why I kept the function
There was a problem hiding this comment.
We agreed to remove this and the tests as it is unused (at least right now)
pkg/newlog/cmd/newlogd.go
Outdated
| if dirName == uploadDevDir { | ||
| if len(filenameFilter.Load().(map[string]any)) != 0 || len(logCounter) != 0 || dedupWindowSize.Load() != 0 { |
There was a problem hiding this comment.
| if dirName == uploadDevDir { | |
| if len(filenameFilter.Load().(map[string]any)) != 0 || len(logCounter) != 0 || dedupWindowSize.Load() != 0 { | |
| if dirName == uploadDevDir && | |
| len(filenameFilter.Load().(map[string]any)) != 0 || | |
| len(logCounter) != 0 || | |
| dedupWindowSize.Load() != 0 { |
There was a problem hiding this comment.
do you think it's better readable?
There was a problem hiding this comment.
Yes, because it does not make the reader expect an else-branch for the inner if.
| continue // we don't care about the error here | ||
| } | ||
| var useEntry bool | ||
| if useEntry = !filterOut(&logEntry); !useEntry { |
There was a problem hiding this comment.
| if useEntry = !filterOut(&logEntry); !useEntry { | |
| if useEntry = filterOut(&logEntry); useEntry { |
There was a problem hiding this comment.
or perhaps directly:
| if useEntry = !filterOut(&logEntry); !useEntry { | |
| if filterOut(&logEntry) { |
There was a problem hiding this comment.
then the variable should be called here dontUseEntry and it breaks the nice flow a little :)
There was a problem hiding this comment.
double negation will break every flow ;-)
also filterOut returns a dontUseEntry not a useEntry. Perhaps filterOut is wrong then?
There was a problem hiding this comment.
also filterOut returns a dontUseEntry not a useEntry
that's exactly why it's negated :)
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
aac93d6 to
71d0a86
Compare
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
71d0a86 to
19061e6
Compare
@eriknordmark the key to solving this issue was to set go version in go.mod file to 1.24.1 (include the patch). Apparently it's a known problem golang/go#62278 (comment) |
|
Folks, I would appreciate if someone could kick off Eden tests! Thank you 😊 |
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
bf52a32 to
0b68040
Compare
|
rebased to see yetus fails no more |
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
0b68040 to
6481b77
Compare
|
I see a set of yetus failures in the summary in https://github.com/lf-edge/eve/actions/runs/14247833424?pr=4702 |
This commit updates the go version to 1.24 and packages eve-api and eve/pkg/pillar to latest versions. Signed-off-by: Paul Gaiduk <[email protected]>
Newlog will have 3 ways to reduce the amount of logs: - filter: filter logs out based on the source code line that produced them - counter: count the number of logs produced by a specific source code line. Add that number to the first occurance of the log and remove the rest - deduplicator (for errors only): record the last X errors in a sliding window and remove duplicates The benchmarking of the newlogd with the new features is in the dedup_test.go file. It shows that CPU and RAM usage increase by a factor of 3 when the features are enabled. So they can be disabled by setting the deduplication window size to 0 and not providing anything to the filter and counter functions. Signed-off-by: Paul Gaiduk <[email protected]>
Fix warnings like "redefinition of the built-in function" in newlogd.go and "should not use dot imports" in newlogd_test.go. Signed-off-by: Paul Gaiduk <[email protected]>
europaul
force-pushed
the
log-filtering-and-dedup
branch
from
6481b77 to
77b1d15
Compare
Done
These will be fixed as a part of a separate effort to make yetus work. |
|
@europaul , the four Eden smoke tests are consistently failing in the Log tests: https://github.com/lf-edge/eve/actions/runs/14266695558/job/40034257107?pr=4702#step:3:1454 Apparently due to timeout, but it requires a investigation to figure out the root cause.... |
Newlog will have 3 ways to reduce the amount of logs:
The benchmarking of the newlogd with the new features is in the dedup_test.go file. It shows that CPU and RAM usage increase by a factor of 3 when the features are enabled. So they can be disabled by setting the deduplication window size to 0 and not providing anything to the filter and counter functions.