Enable Filebeat's Journald input in some docker image variants #7995
Conversation
Filebeat's Journald input requires `journalctl`, so the systemd package is added to all docker image variants.
|
This pull request does not have a backport label. Could you fix it @belimawr? 🙏
|
belimawr
added
Team:Elastic-Agent-Data-Plane
…wr/elastic-agent into 44040-journalctl-on-docker-iamges
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
pierrehilbert
added
the
Team:Elastic-Agent-Control-Plane
|
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane) |
|
@belimawr please add the image size comparison to the description. |
dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl
Outdated
Show resolved
Hide resolved
dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl
Outdated
Show resolved
Hide resolved
|
So I went ahead and found my notes from when we did the initial switch from ubuntu-based to ubi-based images, and I remembered the quirks with journalctl and systemd package. To this end, I built all images with this PR and this is the comparison the images from main, in terms of size: arm64: amd64: Just from the above comparisons, and the security-based reasons, I would say that wolfi-based images should not feature systemd. Now for the ubi-based ones, if the on average 50MB of an increase is prohibitive, we could try out having an intermediate step that installs systemd package and copies the bare minimum of journalctl dependencies to the final image I have to test the above out but AFAIK PS: I would also propose introducing a k8s integration test that tests the journald input 🙂 |
|
IMO based on the above we keep systemd out of the slim, service, and wolfi variants. We would add systemd/journald to the plain |
|
(service has no need for journald/systemd it's the agentless container). |
|
Thanks for the input folks! I believe we should add systemd/journalctl in the slim variant as well as the size increase is very small. How slim does the slim image need to be? Based on @pkoutsovasilis numbers, adding systemd in the slim image is a 30Mb increase on a 1.13Gb, this means 2.65% increase in size, however if a user needs the journald input and migrates from the slim to the ubi variant, that means jumping from 1.13Gb to 1.75Gb, a 54.88% increase, that's a significant change. If the user wants to monitor their Kubernetes cluster, I'd expect them to also want to collect the system logs, and if the logs are on journald, then they will need the journald input. So it looks to be this is a very important use case. Do we have any numbers on this use case and which image was being used before we introduced the need for journalctl in the docker image? In the mean time, I'll look into adding systemd based on the image variant. |
…urnalctl-on-docker-iamges
It needs to be as small as possible while still being useful. That including systemd is a marginal increase is a hint that we need to remove other things not that we can include any arbitrary thing that is 10-50 MB in size because the image is too huge as a starting point. Everyone with this use case is already using the plain elastic-agent image because -slim was only available in 9.0 so we aren't breaking anything. We can put journald in later if people tell us it is critical requirement, but removing it later based on an assumption will be a breaking change if we do it too early. |
…wr/elastic-agent into 44040-journalctl-on-docker-iamges
…urnalctl-on-docker-iamges
belimawr
changed the title
…urnalctl-on-docker-iamges
|
💚 Build Succeeded
History
cc @belimawr |
belimawr
added
backport-8.18
Filebeat's Journald input requires `journalctl`, so the systemd package is added to elastic-agent, elastic-agent-complete, elastic-otel-collector variants. --------- Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit fd33d85) # Conflicts: # dev-tools/packaging/settings.go # dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl
I think that makes sense, if users ask about fixing it in the other versions we can consider going back even further. Though by the same logic, shouldn't 8.18 just have it in the complete image to match 9.0 and this change only lands in 9.1 and 8.19? |
That makes sense. I'll change the backport to 8.18 to only include it in the complete image. And create one to 8.19 with all other images. |
Filebeat's Journald input requires `journalctl`, so the systemd package is added to elastic-agent, elastic-agent-complete, elastic-otel-collector variants. --------- Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit fd33d85) # Conflicts: # dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl
…er image variants (#8492) Filebeat's Journald input requires `journalctl`, so the systemd package is added to elastic-agent, elastic-agent-complete,variants. --------- Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit fd33d85) # Conflicts: # dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl --------- Co-authored-by: Tiago Queiroz <[email protected]>
…r image variants (#8482) * Enable Filebeat's Journald input in some docker image variants (#7995) Filebeat's Journald input requires `journalctl`, so the systemd package is added to elastic-agent, elastic-agent-complete, elastic-otel-collector variants. --------- Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit fd33d85) # Conflicts: # dev-tools/packaging/settings.go # dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl * Fix merge conflicts Mostly re-doing the addition of systemd packages due to the different base images * Update env var usage * Only install jounralctl in the complete variant * remove basic variant from TestKubernetesJournaldInput * remove TestKubernetesJournaldInputOtel as edot collector variant is not relevant to 8.18 * Update changelog/fragments/1745609163-journalctl-on-all-docker-variants.yaml --------- Co-authored-by: Tiago Queiroz <[email protected]> Co-authored-by: Panos Koutsovasilis <[email protected]>
What does this PR do?
Filebeat's Journald input requires
journalctl, so the systemd package is added to all docker image variants.Changes in image size
Images built from
main(f108618):Images built from this PR:
Increase in size
docker.elastic.co/elastic-agent/elastic-agent-complete: 10MBdocker.elastic.co/elastic-agent/elastic-otel-collector: 50MBdocker.elastic.co/elastic-agent/elastic-agent: 40MBWhy is it important?
Currently it is only possible to use integrations that requires the journald input with the complete docker image variant, this PR allows it to be used with any docker image variant. The complete docker image is the largest with more than 3Gb, while the standard Elastic-Agent image is less than 2Gb.
Checklist
I have commented my code, particularly in hard-to-understand areasI have made corresponding change to the default configuration filesI have added tests that prove my fix is effective or that my feature works./changelog/fragmentsusing the changelog toolI have added an integration test or an E2E test## Disruptive User ImpactAuthor's Checklist
How to test this PR locally
journalctlon each docker image by running the following script/commands:Related issues
journalctlto Filebeat and Elastic-Agent docker images beats#44040Questions to ask yourself