CNDB-15508: Query planner metrics

[pkolaczk]

This commit adds new metrics related to the operation of SAI query
planner. The metrics should help checking if the query planner makes
proper decisions by correlating them with the other metrics,
e.g. the metrics of the actual query execution.

Per-query metrics (histograms):

• RowsEstimated: the estimated number of rows to be returned by
  the query
• CostEstimated: the abstract cost of query execution
• InverseSelectivityEstimated: the inverse of query selectivity,
  before applying the query LIMIT
  (1 means the query selects all rows, 10 means it
  selects every 10th row, etc.)
• IndexReferencesInQuery: the number of index references in the
  unoptimized query execution plan (the same index may
  be referenced multiple times and counts separately)
• IndexReferencesInPlan: the number of index references in the
  optimized query execution plan (the same index may
  be referenced multiple times and counts separately)

Per-table:

• TotalRowsEstimated: counts the sum of all row estimates from
  all completed queries
• TotalCostEstimated: counts the sum of all cost estimates from
  all completed queries
• TotalQueriesCompletedInSelectivityGroup{N}, where N in [0, 12):
  counts the number of completed queries with selectivity S:
  10^(-N-1) < S <= 10^(-N) for N < 11,
  S <= 10^(-N) for N = 11
  In other words, the higher the group N, the smaller fraction of rows
  the query is estimated to return. The selectivity calculation
  does not include the final LIMIT of the query.

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[ekaterinadimitrova2]

Not a full review, just started looking into this, just dropped a few quick comments

[ekaterinadimitrova2]

Looks great, left just some questions.
There are some GH problems today and CI fails. You will need to restart it to check the results.

[adelapena]

indexReferencesInQuery and indexReferencesInPlan have associated histograms in TableQueryMetrics, but not counters. searchExecutedBeforeOrder and filterExecutedAfterOrderedScan have counters but not histograms. Is that intentional?

[ekaterinadimitrova2]

While it breaks the consistency, I am wondering whether it is this way because:

• histograms for boolean values are not very useful
• for the others the index reference counts are small integers
  Maybe we add some note to make clear the rationale? Wondering whether that would be helpful or it may become outdated doc in time

[pkolaczk]

As for booleans - I think nothing technically stops us from having a boolean histogram (but we'd have to convert it to 0 and 1 as currently histograms are based on longs); but I feel it wouldn't be very useful in this case. We get 100% of the same information from the per-table counters.

As for the other histograms - well, it is hard to represent them accurately as per table counters. Like, I feel it doesn't make much sense to sum all index references from all queries... So we have counters in per table for things which sum events naturally (like rows returned - it is quite easy to understand what a total number of rows returned is), but per-query histograms for other properties of queries which are not "summable" in that way.

Note there are many other histograms in perQuery which don't have (a) corresponding PerTable counter(s), so I don't think it breaks consistency. Anyway, we really need to expose those histograms in CNDB somehow, but let's live it for a followup ticket.

[adelapena]

Makes perfect sense to me, thanks.

[ekaterinadimitrova2]

Overall looks great, I just left some tiny comments.
There is some issue with CI so it did not run.
Regarding testing - shouldn't we add some test for the new flag added - cassandra.sai.metrics.query_plan.enabled?

[ekaterinadimitrova2]

While it breaks the consistency, I am wondering whether it is this way because:

• histograms for boolean values are not very useful
• for the others the index reference counts are small integers
  Maybe we add some note to make clear the rationale? Wondering whether that would be helpful or it may become outdated doc in time

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-2130 rejected by Butler

---

3 regressions found
See build details here

---

Found 3 new test failures

Test	Explanation	Runs	Upstream
o.a.c.index.sai.cql.VectorCompaction100dTest.testOneToManyCompaction[eb false]	REGRESSION	🔴⚪	0 / 18
o.a.c.index.sai.cql.VectorKeyRestrictedOnPartitionTest.partitionRestrictedWidePartitionBqCompressedTest[db false] (compression)	REGRESSION	🔵🔴	0 / 18
o.a.c.index.sai.cql.VectorSiftSmallTest.testCompaction[db false]	REGRESSION	🔴⚪	0 / 18

---

Found 3 known test failures

[adelapena]

Not a big deal, but Math.log10 is a relatively costly operation, and we'll call it seven times per query if per-kind metrics are enabled. Since this is the only read of QueryPlanInfo.selectivityEstimated, maybe we can put the logarithm of the selectivity in the plan info, as QueryPlanInfo.logSelectivityEstimated, so it's computed only once per query?

[adelapena]

Perhaps we could do the same for the quicker calls to Math.round for costEstimated, rowsToReturnEstimated, rowsToFetchEstimated and keysToIterateEstimated? Those will be repeated 14 times per query with per-query-kind metrics, so we can save up to 5*13=65 calls per query:

public final long costEstimated;
public final long rowsToReturnEstimated;
public final long rowsToFetchEstimated;
public final long keysToIterateEstimated;
public final int logSelectivityEstimated;
...
public PlanInfo(@Nonnull Plan.RowsIteration originalPlan, @Nonnull Plan.RowsIteration optimizedPlan)
{
    this.costEstimated = Math.round(optimizedPlan.fullCost());
    this.rowsToReturnEstimated = Math.round(optimizedPlan.expectedRows());
    this.rowsToFetchEstimated = Math.round(optimizedPlan.estimatedRowsToFetch());
    this.keysToIterateEstimated = Math.round(optimizedPlan.estimatedKeysToIterate());
    this.logSelectivityEstimated = (Math.min(20, (int) Math.floor(-Math.log10(optimizedPlan.selectivity()))));

That was the intention of having a QueryContext.Snapshot class, reducing the amplification caused by having multiple consumers of these metrics (per-query-kind metrics and slow query logger).