Fix prometheus metric format (#2825) #2899

ZhengweiZhu · 2025-02-23T13:08:12Z

Fix PrometheusMetricsDumper to dump unique comment of a certain metric name to follow the prometheus specification.
Also refine the prometheus ut case to support mvar output.

What problem does this PR solve?

Issue Number:
#2825

Problem Summary:

What is changed and the side effects?

Changed:

Side effects:

Performance effects(性能影响): NO
Breaking backward compatibility(向后兼容性):

Check List:

Please make sure your changes are compilable(请确保你的更改可以通过编译).
When providing us with a new feature, it is best to add related tests(如果你向我们增加一个新的功能, 请添加相关测试).
Please follow Contributor Covenant Code of Conduct.(请遵循贡献者准则).

wasphin · 2025-02-23T15:13:03Z

src/brpc/builtin/prometheus_metrics_service.cpp

-         << name << " " << desc << '\n';
+    // To meet specification of prometheus metric, there should be only
+    // one TYPE field for the same metric name.
+    if (metrics_name.as_string() != _last_metric_name) {


Simply ignoring metrics that are duplicates of the previous ones does not completely solve the problem. If the metrics themselves are repetitive, perhaps we should try to identify and address the root cause; if it's just a matter of duplicate names, consider fixing the metrics name.

Let me add some background knowledge. The metrics are stored in butil::FlatMap and the uniqueness of metric name is guaranteed. The problem arises from the introduction of mvar, where one metric name will correspond to multiple metric with different labels. My patch just handles this problem

How about adding a virtual function for mvabr to Dumper? I think this can completely solve the problem.

bool mutil_dump( const std::string& mutil_name, const std::vector<std::string>& names, const std::vector<std::string>& descriptions) override;

@chenBright @wasphin The pr is updated according to review comment and ut case is added. Please take a look and merge if nothing wrong.

OK. I'll take a look at the changes in the next few days.

wwbmmm · 2025-02-24T02:46:27Z

This PR is similar to #2870, which one is better?

ZhengweiZhu · 2025-02-24T03:29:47Z

This PR is similar to #2870, which one is better?

The #2870 introduced a std::set container to remove duplicates, which is unnecessary and impacts performance. As the metrics are stored in butil::FlatMap and the uniqueness of metric name is guaranteed.

When we just expose bvar, the current upstream code is fine. The problem arises from the introduction of mvar, where one metric name will correspond to multiple metric with different labels. My patch handles this problem and follow the specifition of Prometheus metric.

wwbmmm · 2025-02-24T04:25:25Z

LGTM

wasphin · 2025-02-24T05:52:57Z

There's a new dump_comment API when introducing mvar by @serverglen, is this the case to set some HELP/TYPE for it? dump_comment is only used by mvar.

$ git grep dump_comment

src/bvar/multi_dimension_inl.h:242:    if (label_names.empty() || !dumper->dump_comment(name(), METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:279:        if (!dumper->dump_comment(name() + "_latency", METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:312:        if (!dumper->dump_comment(name() + "_max_latency", METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:323:        if (!dumper->dump_comment(name() + "_qps", METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:334:        if (!dumper->dump_comment(name() + "_count", METRIC_TYPE_COUNTER)) {
src/bvar/variable.h:56:    virtual bool dump_comment(const std::string&, const std::string& /*type*/) {

ZhengweiZhu · 2025-02-24T09:44:12Z

There's a new dump_comment API when introducing mvar by @serverglen, is this the case to set some HELP/TYPE for it? dump_comment is only used by mvar.

$ git grep dump_comment

src/bvar/multi_dimension_inl.h:242:    if (label_names.empty() || !dumper->dump_comment(name(), METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:279:        if (!dumper->dump_comment(name() + "_latency", METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:312:        if (!dumper->dump_comment(name() + "_max_latency", METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:323:        if (!dumper->dump_comment(name() + "_qps", METRIC_TYPE_GAUGE)) {
src/bvar/multi_dimension_inl.h:334:        if (!dumper->dump_comment(name() + "_count", METRIC_TYPE_COUNTER)) {
src/bvar/variable.h:56:    virtual bool dump_comment(const std::string&, const std::string& /*type*/) {

Reasonable. By overriding dump_comment method of PrometheusMetricsDumper to output HELP/TYPE for a metric, then the dump method should be changed to only output name and value (not including HELP/TYPE). I will try it this way.

ZhengweiZhu · 2025-03-01T09:20:19Z

Update: The original patch only handles mvar not in LatencyRecorder type correctly. When the mvar is in LatencyRecorder type, things get more complicated.

For example, we expose a mvar in LatencyRecorder type with two different labels.

    bvar::MultiDimension<bvar::LatencyRecorder > my_lat("dump_lat", {"idc", "method", "status"});
    bvar::LatencyRecorder* lat1 = my_lat.get_stats({"gz", "post", "200"});
    *lat1 << 1;
    *lat1 << 2;

    bvar::LatencyRecorder* lat2 = my_lat.get_stats({"tj", "post", "300"});
    *lat2 << 3;
    *lat2 << 4;

For the upstream code

#curl -s 127.0.0.1:8000/brpc_metrics

# HELP dump_lat_latency
# TYPE dump_lat_latency gauge
dump_lat_latency{idc="gz",method="post",status="200",quantile="0"} 0
# HELP dump_lat_latency
# TYPE dump_lat_latency gauge
dump_lat_latency{idc="gz",method="post",status="200",quantile="80"} 0
...

#curl -s 127.0.0.1:8000/brpc_metrics | promtool check metrics
error while linting: text format parsing error in line 5: second TYPE line for metric name "dump_lat_latency", or TYPE reported after samples

After applying my original patch (judging if last metric name is equal to this one)

#curl -s 127.0.0.1:8000/brpc_metrics

# HELP dump_lat_latency
# TYPE dump_lat_latency gauge
dump_lat_latency{idc="gz",method="post",status="200",quantile="0"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="80"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="90"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="99"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="999"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="9999"} 0
# HELP dump_lat_max_latency
# TYPE dump_lat_max_latency gauge
dump_lat_max_latency{idc="gz",method="post",status="200"} 0
# HELP dump_lat_qps
# TYPE dump_lat_qps gauge
dump_lat_qps{idc="gz",method="post",status="200"} 0
# HELP dump_lat_count
# TYPE dump_lat_count gauge
dump_lat_count{idc="gz",method="post",status="200"} 2
# HELP dump_lat_latency
# TYPE dump_lat_latency gauge
dump_lat_latency{idc="tj",method="post",status="300",quantile="0"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="80"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="90"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="99"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="999"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="9999"} 0
# HELP dump_lat_max_latency
# TYPE dump_lat_max_latency gauge
dump_lat_max_latency{idc="tj",method="post",status="300"} 0
# HELP dump_lat_qps
# TYPE dump_lat_qps gauge
dump_lat_qps{idc="tj",method="post",status="300"} 0
# HELP dump_lat_count
# TYPE dump_lat_count gauge
dump_lat_count{idc="tj",method="post",status="300"} 2

#curl -s 127.0.0.1:8000/brpc_metrics | promtool check metrics
error while linting: text format parsing error in line 19: second TYPE line for metric name "dump_lat_latency", or TYPE reported after samples
The problem here is that the second "# TYPE dump_lat_latency gauge" in line 19 is equal to the one in line 2.

So we need to refactor code and rearrange the output of mvar in LatencyRecorder type by putting the same metric name (with different labels) together, such like the following(which pass the promtool check):

# HELP dump_lat_latency
# TYPE dump_lat_latency gauge
dump_lat_latency{idc="gz",method="post",status="200",quantile="0"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="80"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="90"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="99"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="999"} 0
dump_lat_latency{idc="gz",method="post",status="200",quantile="9999"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="0"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="80"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="90"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="99"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="999"} 0
dump_lat_latency{idc="tj",method="post",status="300",quantile="9999"} 0
# HELP dump_lat_max_latency
# TYPE dump_lat_max_latency gauge
dump_lat_max_latency{idc="gz",method="post",status="200"} 0
dump_lat_max_latency{idc="tj",method="post",status="300"} 0
# HELP dump_lat_qps
# TYPE dump_lat_qps gauge
dump_lat_qps{idc="gz",method="post",status="200"} 0
dump_lat_qps{idc="tj",method="post",status="300"} 0
# HELP dump_lat_count
# TYPE dump_lat_count gauge
dump_lat_count{idc="gz",method="post",status="200"} 2
dump_lat_count{idc="tj",method="post",status="300"} 2

Fix PrometheusMetricsDumper to dump unique comment of a certain metric name to follow the prometheus specification. Also refine the prometheus ut case to support mvar output

ZhengweiZhu · 2025-03-07T05:46:32Z

@chenBright There's an ut failure BthreadTest.trace which seems to be brought in by you recent commit?

src/bvar/multi_dimension_inl.h

chenBright

LGTM

cc @wasphin

wasphin

LGTM

wasphin reviewed Feb 23, 2025

View reviewed changes

ZhengweiZhu marked this pull request as draft March 3, 2025 02:38

ZhengweiZhu force-pushed the fixPrometheusFmt branch 2 times, most recently from cb80cf6 to 14e911a Compare March 5, 2025 10:38

ZhengweiZhu marked this pull request as ready for review March 5, 2025 10:44

Fix format of prometheus service (apache#2825)

65c417a

Fix PrometheusMetricsDumper to dump unique comment of a certain metric name to follow the prometheus specification. Also refine the prometheus ut case to support mvar output

ZhengweiZhu force-pushed the fixPrometheusFmt branch from 14e911a to 65c417a Compare March 5, 2025 12:51

ZhengweiZhu requested a review from chenBright March 5, 2025 13:40

chenBright reviewed Mar 10, 2025

View reviewed changes

src/bvar/multi_dimension_inl.h Show resolved Hide resolved

chenBright approved these changes Mar 11, 2025

View reviewed changes

ZhengweiZhu requested a review from wasphin March 11, 2025 12:06

wasphin approved these changes Mar 11, 2025

View reviewed changes

chenBright linked an issue Mar 12, 2025 that may be closed by this pull request

brpc的brpc_metrics的输出格式与prometheus的标准不符，尤其MultiDimension<bvar::LatencyRecorder>方式 #2825

Closed

chenBright merged commit b4ce61f into apache:master Mar 12, 2025
20 checks passed

chenBright mentioned this pull request Mar 12, 2025

Fix/metric emission tracking #2825 #2870

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix prometheus metric format (#2825) #2899

Fix prometheus metric format (#2825) #2899

ZhengweiZhu commented Feb 23, 2025 •

edited

Loading

wasphin Feb 23, 2025

ZhengweiZhu Feb 24, 2025

chenBright Mar 3, 2025

ZhengweiZhu Mar 5, 2025

chenBright Mar 7, 2025

wwbmmm commented Feb 24, 2025

ZhengweiZhu commented Feb 24, 2025 •

edited

Loading

wwbmmm commented Feb 24, 2025

wasphin commented Feb 24, 2025

ZhengweiZhu commented Feb 24, 2025 •

edited

Loading

ZhengweiZhu commented Mar 1, 2025 •

edited

Loading

ZhengweiZhu commented Mar 7, 2025

chenBright left a comment •

edited

Loading

wasphin left a comment

Fix prometheus metric format (#2825) #2899

Fix prometheus metric format (#2825) #2899

Conversation

ZhengweiZhu commented Feb 23, 2025 • edited Loading

What problem does this PR solve?

What is changed and the side effects?

Check List:

wasphin Feb 23, 2025

Choose a reason for hiding this comment

ZhengweiZhu Feb 24, 2025

Choose a reason for hiding this comment

chenBright Mar 3, 2025

Choose a reason for hiding this comment

ZhengweiZhu Mar 5, 2025

Choose a reason for hiding this comment

chenBright Mar 7, 2025

Choose a reason for hiding this comment

wwbmmm commented Feb 24, 2025

ZhengweiZhu commented Feb 24, 2025 • edited Loading

wwbmmm commented Feb 24, 2025

wasphin commented Feb 24, 2025

ZhengweiZhu commented Feb 24, 2025 • edited Loading

ZhengweiZhu commented Mar 1, 2025 • edited Loading

For the upstream code

After applying my original patch (judging if last metric name is equal to this one)

ZhengweiZhu commented Mar 7, 2025

chenBright left a comment • edited Loading

Choose a reason for hiding this comment

wasphin left a comment

Choose a reason for hiding this comment

ZhengweiZhu commented Feb 23, 2025 •

edited

Loading

ZhengweiZhu commented Feb 24, 2025 •

edited

Loading

ZhengweiZhu commented Feb 24, 2025 •

edited

Loading

ZhengweiZhu commented Mar 1, 2025 •

edited

Loading

chenBright left a comment •

edited

Loading