Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION]:ecs上采集日志后用日志时间替换采集时间如何配置 #1990

Closed
zhangchk opened this issue Dec 24, 2024 · 0 comments
Closed
Labels
question Further information is requested

Comments

@zhangchk
Copy link

zhangchk commented Dec 24, 2024

背景

采集流程:ecs-->>ilogtail-->>loki-->>grafana查看
ilogtail版本:2.0.0
loki版本:2.7.3

期望

将采集时间,替换为日志中的打印时间,以时间戳形式保留(毫秒时间戳)

日志样例

2024-12-24 11:25:24.083 INFO  [TID:828a47d610594ef083d7b29e5.191.7350107238612947] [DubboServerHandler-172.16.2.110:20880-thread-234] [20241224112523093436B9589657656DB42C] [] [] [c.f.d.e.p.s.DataCollectServiceImpl collect 45 : collect param: SendCollectEventParam(billNo=MEDUOF209136, containerNo=null, portCode=CNNGB, isExport=E)

版本一

processors配置截取如下,原本processor_regex是processor_parse_regex_native,但是processor_parse_regex_native、processor_parse_timestamp_native添加后都有报错如下

processors:
  - Type: processor_split_log_regex
    SplitRegex: \d+-\d+-\d+\s\d+:\d+:\d+\s.*
    SplitKey: content
    PreserveOthers: true
  - Type: processor_regex
    SourceKey: content
    Regex: (\S+ \S+) (.*)
    Keys:
      - time
      - msg
  - Type: processor_parse_timestamp_native
    SourceKey: time
    SourceFormat: '%Y-%m-%d %H:%M:%S'

logtail_plugin.LOG报错

2024-12-24 10:55:36 [INF] [plugin_export.go:179] [HoldOn] Hold on:start flag:1
2024-12-24 10:55:36 [INF] [plugin_manager.go:150] [func1] [dcms-expose/2]       Stop config in goroutine:begin
2024-12-24 10:55:36 [INF] [logstore_config.go:172] [Stop] [dcms-expose/2]       config stop:begin       exit:true
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:389] [Stop] [dcms-expose/2]      metric plugins stop:done        service plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:392] [Stop] [dcms-expose/2]      processor plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:395] [Stop] [dcms-expose/2]      aggregator plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:418] [Stop] [dcms-expose/2]      flusher plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:427] [Stop] [dcms-expose/2]      extension plugins stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:176] [Stop] [dcms-expose/2]       Plugin Runner stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:179] [Stop] [dcms-expose/2]       config stop:success
2024-12-24 10:55:36 [INF] [plugin_manager.go:155] [func1] [dcms-expose/2]       Stop config in goroutine:end
2024-12-24 10:55:36 [INF] [logstore_config.go:172] [Stop] [shennong_log_profile,logtail_plugin_profile] config stop:begin       exit:true
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:389] [Stop] [shennong_log_profile,logtail_plugin_profile]        metric plugins stop:done        service plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:392] [Stop] [shennong_log_profile,logtail_plugin_profile]        processor plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:395] [Stop] [shennong_log_profile,logtail_plugin_profile]        aggregator plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:418] [Stop] [shennong_log_profile,logtail_plugin_profile]        flusher plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:427] [Stop] [shennong_log_profile,logtail_plugin_profile]        extension plugins stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:176] [Stop] [shennong_log_profile,logtail_plugin_profile] Plugin Runner stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:179] [Stop] [shennong_log_profile,logtail_plugin_profile] config stop:success
2024-12-24 10:55:36 [INF] [logstore_config.go:172] [Stop] [logtail_alarm,logtail_alarm] config stop:begin       exit:true
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:389] [Stop] [logtail_alarm,logtail_alarm]        metric plugins stop:done        service plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:392] [Stop] [logtail_alarm,logtail_alarm]        processor plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:395] [Stop] [logtail_alarm,logtail_alarm]        aggregator plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:418] [Stop] [logtail_alarm,logtail_alarm]        flusher plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:427] [Stop] [logtail_alarm,logtail_alarm]        extension plugins stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:176] [Stop] [logtail_alarm,logtail_alarm] Plugin Runner stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:179] [Stop] [logtail_alarm,logtail_alarm] config stop:success
2024-12-24 10:55:36 [INF] [logstore_config.go:172] [Stop] [logtail_containers,logtail_containers]       config stop:begin       exit:true
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:389] [Stop] [logtail_containers,logtail_containers]      metric plugins stop:done        service plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:392] [Stop] [logtail_containers,logtail_containers]      processor plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:395] [Stop] [logtail_containers,logtail_containers]      aggregator plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:418] [Stop] [logtail_containers,logtail_containers]      flusher plugins stop:done
2024-12-24 10:55:36 [INF] [plugin_runner_v1.go:427] [Stop] [logtail_containers,logtail_containers]      extension plugins stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:176] [Stop] [logtail_containers,logtail_containers]       Plugin Runner stop:done
2024-12-24 10:55:36 [INF] [logstore_config.go:179] [Stop] [logtail_containers,logtail_containers]       config stop:success
2024-12-24 10:55:36 [INF] [checkpoint_manager.go:108] [HoldOn] checkpoint:HoldOn
2024-12-24 10:55:36 [INF] [checkpoint_manager.go:128] [run] checkpoint:HoldOn success
2024-12-24 10:55:36 [INF] [plugin_export.go:187] [HoldOn] Hold on:success
2024-12-24 10:55:36 [INF] [plugin_export.go:189] [HoldOn] logger:close and recover
2024-12-24 10:55:36 [ERR] [logger.go:376] [1] AlarmType:CATCH_STANDARD_OUTPUT_ALARM     err:read |0: file already closed
2024-12-24 10:55:36 [ERR] [logger.go:376] [1] AlarmType:CATCH_STANDARD_OUTPUT_ALARM     err:read |0: file already closed

版本二

参考[ilogtail time是unix时间戳 是否支持改为yyyy-mm-dd格式 · alibaba/loongcollector · Discussion #1196 · GitHub](#1196)

修改配置如下

processors:
  - Type: processor_split_log_regex
    SplitRegex: \d+-\d+-\d+\s\d+:\d+:\d+\s.*
    SplitKey: content
    PreserveOthers: true
  - Type: processor_regex
    SourceKey: content
    Regex: (\S+ \S+) (.*)
    Keys:
      - log_time
      - msg
  - Type: processor_strptime
    SourceKey: log_time
    SourceFormat: '%Y-%m-%d %H:%M:%S.%f'
    EnablePreciseTimestamp: true

ilogtail启动后不报错,但是日志没有采集至loki

请求帮助

  1. 版本一中的配置为什么不行?是按照官网配置的

  2. 版本二中为什么采集不到日志至loki?

@zhangchk zhangchk added the question Further information is requested label Dec 24, 2024
@zhangchk zhangchk changed the title [QUESTION]:ecs上使用原生插件后无法采集日志 [QUESTION]:ecs上采集日志后用日志时间替换采集时间如何配置 Dec 24, 2024
@zhangchk zhangchk closed this as completed Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant