[Celerdata] Add a grok expression to parse the new log format of BE

Signed-off-by: yandongxiao <[email protected]>
DataDog · Feb 12, 2025 · ae6abfb · ae6abfb
1 parent add3079
commit ae6abfb
Show file tree

Hide file tree

Showing 4 changed files with 11 additions and 2 deletions.
diff --git a/celerdata/CHANGELOG.md b/celerdata/CHANGELOG.md
@@ -1,5 +1,11 @@
 # CHANGELOG - celerdata
 
+## 1.0.1 / 2025-02-12
+
+***Added***:
+
+* Add a grok expression to parse the new log format of BE
+
 ## 1.0.0 / 2024-03-11
 
 ***Added***:

diff --git a/celerdata/README.md b/celerdata/README.md
@@ -18,7 +18,7 @@ To collect StarRocks [metrics][9] and logs:
 2. Install the CelerData check on your host with the following command:
 
    ```shell
-   datadog-agent integration install -t datadog-celerdata==1.0.0
+   datadog-agent integration install -t datadog-celerdata==1.0.1
    ```
 
 ### Configuration

diff --git a/celerdata/assets/logs/celerdata.yaml b/celerdata/assets/logs/celerdata.yaml
@@ -18,6 +18,7 @@ pipeline:
         - 2024-03-28 18:58:43,380 INFO (nioEventLoopGroup-4-3|154) [RestBaseAction.handleRequest():73] receive http request. url=/api/health
         - 2024-04-10 10:32:23.937+08:00 INFO (stateChangeExecutor|95) [StateChangeExecutor.runOneCycle():179] finished to transfer FE type from INIT to LEADER
         - I0328 10:22:09.303526 1066 olap_server.cpp:844] try to perform path gc by tablet!
+        - I20250211 13:17:55.023299 127358726047296 query_context.cpp:60] finished query_id:837ca8a5-e8a4-11ef-89f3-0e809d90bae3 context life time:1476324 cpu costs:428108 peak memusage:298504 scan_bytes:5139 spilled bytes:0
       grok:
         supportRules: fe_other_log %{word:loglevel} \(%{notSpace}\|%{integer}\) \[%{word:class_name}.%{word:method_name}\(\):%{integer:file_line}\] %{data:message}
         matchRules: >-
@@ -26,6 +27,8 @@ pipeline:
           fe.log2 %{date("yyyy-MM-dd' 'HH:mm:ss.SSSZZ"):timestamp} %{fe_other_log}
 
           be.info %{regex("\\w"):loglevel}%{date("MMdd HH:mm:ss.SSSSSS"):timestamp}\s+%{integer:thread_id} %{notSpace:file_name}\:%{integer:file_line}\] %{data:message}
+
+          be.info2 (?<loglevel>[A-Z])%{date("yyyyMMdd HH:mm:ss.SSSSSS"):timestamp}\s+%{integer:thread_id}\s+%{notSpace:file_name}:%{integer:file_line}\]\s+%{data:message}
     - type: status-remapper
       name: Define `loglevel` as the official status of the log
       enabled: true

diff --git a/celerdata/datadog_checks/celerdata/__about__.py b/celerdata/datadog_checks/celerdata/__about__.py
@@ -1 +1 @@
-__version__ = '1.0.0'
+__version__ = '1.0.1'