Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION]: input_file插件采集容器日志,没有携带metadata信息 #1814

Closed
PalanQu opened this issue Oct 17, 2024 · 7 comments
Closed
Labels
question Further information is requested

Comments

@PalanQu
Copy link

PalanQu commented Oct 17, 2024

ilogtail通过daemonset部署,版本commit为d413393c79eae41c95bfce03b1e2c519372892ba,最后一个commit为2024年9月15日,采集配置如下,有两个问题

inputs:
  - Type: input_file
    FilePaths:
    - /log/*.info
    EnableContainerDiscovery: true
    AppendingLogPositionMeta: true
    AllowingIncludedByMultiConfigs: true
    CollectingContainersMeta: true
    ExternalK8sLabelTag:
      cluster: subApplication
    ContainerFilters: 
      IncludeK8sLabel:
        cluster: "^.*$"
  1. 采集到的日志并没有携带metadata信息,例如采集到的日志如下
2024-10-17 19:44:21 {"__file_offset__":"4087590","logtime":"2024-10-17T19:44:20.109445","level":"INFO","logger":"src/allocator/free_space_syncer.cpp:119","thread":"00000085","bthread":"0000000100000916","xrayTraceId":"FACAFA77886CDEBA","msg":"sync free space info success, node num = 24","_prdline_":"base","expireDay":"14","__time__":"1729165460"}
  1. ExternalK8sLabelTag没有按照预期将label中cluster放到subApplication字段
@PalanQu PalanQu added the question Further information is requested label Oct 17, 2024
@Takuka0311
Copy link
Collaborator

后续使用release版本可行吗?

@PalanQu
Copy link
Author

PalanQu commented Nov 8, 2024

@Takuka0311 抱歉回复晚了,我使用2.0.7还是一样的问题,没有metadata

@Takuka0311
Copy link
Collaborator

@Takuka0311 抱歉回复晚了,我使用2.0.7还是一样的问题,没有metadata

目前怀疑是输出的时候,协议有些问题。在2.0之前,所有k8smeta相关的信息都是在log的content里,在任意flusher处输出都能看到;2.0版本之后,k8smeta相关都在log的tag里,输出到非sls的目标需要转换协议搞一下。
image

协议转换:https://ilogtail.gitbook.io/ilogtail-docs/developer-guide/log-protocol/converter

目前具体的使用场景可以看HTTP flusher的Convert.Protocol参数:https://ilogtail.gitbook.io/ilogtail-docs/plugins/flusher/flusher-http

@PalanQu
Copy link
Author

PalanQu commented Nov 11, 2024

感谢回复,我学习一下

@Takuka0311
Copy link
Collaborator

@Takuka0311 抱歉回复晚了,我使用2.0.7还是一样的问题,没有metadata

目前怀疑是输出的时候,协议有些问题。在2.0之前,所有k8smeta相关的信息都是在log的content里,在任意flusher处输出都能看到;2.0版本之后,k8smeta相关都在log的tag里,输出到非sls的目标需要转换协议搞一下。 image

协议转换:https://ilogtail.gitbook.io/ilogtail-docs/developer-guide/log-protocol/converter

目前具体的使用场景可以看HTTP flusher的Convert.Protocol参数:https://ilogtail.gitbook.io/ilogtail-docs/plugins/flusher/flusher-http

我进行了对比验证,基本可以确认是这个问题了。我配置了两个采集配置,仅flusher不同

下面是flusher_stdout的输出结果:
image

下面是flusher_sls的输出结果:
image

@Takuka0311
Copy link
Collaborator

您现在用的flusher是?如果没有适配Convert,可以参考flusher_kafka_v2、flusher_http实现一下

@PalanQu
Copy link
Author

PalanQu commented Nov 12, 2024

@Takuka0311 测试已经成功,非常感谢

@PalanQu PalanQu closed this as completed Nov 12, 2024
@alibaba alibaba locked and limited conversation to collaborators Nov 12, 2024
@messixukejia messixukejia converted this issue into discussion #1880 Nov 12, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants