Fix Timeseries/recent endpoint speed

[rohaan-rma]

Fixes #1078
Proposed query update that changes a single Timeseries/Recent request from 30 seconds down to half a second on our SWT db export.

WITH base_ids AS (
    SELECT DISTINCT
        ( ts_code ),
        cwms_ts_id
    FROM
        av_cwms_ts_id2
    WHERE
        cwms_ts_id IN ( 'HAMM.Stage-Alt.Inst.15Minutes.0.Decodes-Raw', 'HAMM.Stage.Inst.15Minutes.0.Decodes-Raw' )
), av_tsv_limited AS (
    SELECT
        *
    FROM
        at_tsv_2023
    WHERE
        date_time BETWEEN DATE '2024-06-24' AND DATE '2024-07-22'
        AND ts_code IN (
            SELECT
                ts_code
            FROM
                base_ids
        )
    UNION ALL
    SELECT
        *
    FROM
        at_tsv_2024
    WHERE
        date_time BETWEEN DATE '2024-06-24' AND DATE '2024-07-22'
        AND ts_code IN (
            SELECT
                ts_code
            FROM
                base_ids
        )
), max_values AS (
    SELECT
        b.cwms_ts_id,
        t.ts_code,
        t.date_time,
        t.value,
        t.version_date,
        t.data_entry_date,
        t.quality_code,
        ex.earliest_entry_time       start_date,
        ex.latest_entry_time         end_date,
        MAX(t.date_time)
        OVER(PARTITION BY t.ts_code) AS max_date_time
    FROM
             av_tsv_limited t
        JOIN base_ids      b ON b.ts_code = t.ts_code
        JOIN at_ts_extents ex ON ex.ts_code = t.ts_code
                                 AND ex.version_time = t.version_date
)
SELECT
    max_values.cwms_ts_id,
    max_values.date_time,
    max_values.version_date,
    max_values.data_entry_date,
    max_values.quality_code,
    max_values.start_date,
    max_values.end_date,
    cwms_util.get_default_units(
        cwms_ts.get_base_parameter_id(max_values.ts_code),
        'EN'
    )                    AS def_units,
    max_values.date_time AS max_date_time,
    cwms_util.convert_units(max_values.value,
                            cwms_util.get_default_units(
                      cwms_ts.get_base_parameter_id(max_values.ts_code),
                      'SI'
                  ),
                            cwms_util.get_default_units(
                      cwms_ts.get_base_parameter_id(max_values.ts_code),
                      'EN'
                  ))                   AS value_at_max_date
FROM
    max_values
WHERE
    max_values.date_time = max_values.max_date_time;

[MikeNeilson]

Looks reasonable. Definitely need to chew on it a bit. Fairly complex query, though I doubt unnecessarily so.

[MikeNeilson]

These should be AT_TSV_PREV_YEAR and AT_TSV_CURRENT_YEAR and calculated at usage. Otherwise we'll have to remember to update them constantly.

[rohaan-rma]

Yup you're correct. I just had those as place holders for now.

[rohaan-rma]

Newly fixed proposed DB query:

• Fixed unit conversion (Added unit_id from AV_CWMS_TS_ID2)
• Added office-id to the where condition (Adds office-id only if not null)

with
  "base_ids" as (
    select distinct "CWMS_20"."AV_CWMS_TS_ID2"."TS_CODE", "CWMS_20"."AV_CWMS_TS_ID2"."CWMS_TS_ID", "CWMS_20"."AV_CWMS_TS_ID2"."UNIT_ID"
    from "CWMS_20"."AV_CWMS_TS_ID2"
    where (
      "CWMS_20"."AV_CWMS_TS_ID2"."CWMS_TS_ID" in ('HAMM.Stage.Inst.15Minutes.0.Decodes-Raw')
      and "CWMS_20"."AV_CWMS_TS_ID2"."DB_OFFICE_ID" = 'SWT'
    )
  ),
  "tsv_limited" as (
    select *
    from "CWMS_20"."AT_TSV_2023"
    where (
      "AT_TSV_2023"."DATE_TIME" between date '2024-06-24' and date '2024-07-22'
      and "AT_TSV_2023"."TS_CODE" in (
        select "base_ids"."TS_CODE"
        from "base_ids"
      )
    )
    union all
    select *
    from "CWMS_20"."AT_TSV_2024"
    where (
      "AT_TSV_2024"."DATE_TIME" between date '2024-06-24' and date '2024-07-22'
      and "AT_TSV_2024"."TS_CODE" in (
        select "base_ids"."TS_CODE"
        from "base_ids"
      )
    )
  ),
  "max_values" as (
    select
      "base_ids"."CWMS_TS_ID",
      "base_ids"."UNIT_ID",
      "tsv_limited"."TS_CODE",
      "tsv_limited"."DATE_TIME",
      "tsv_limited"."VALUE",
      "tsv_limited"."VERSION_DATE",
      "tsv_limited"."DATA_ENTRY_DATE",
      "tsv_limited"."QUALITY_CODE",
      "CWMS_20"."AT_TS_EXTENTS"."EARLIEST_ENTRY_TIME" "START_DATE",
      "CWMS_20"."AT_TS_EXTENTS"."LATEST_ENTRY_TIME" "END_DATE",
      max("tsv_limited"."DATE_TIME") over (partition by "tsv_limited"."TS_CODE") "max_date_time"
    from "tsv_limited"
      join "base_ids"
        on "base_ids"."TS_CODE" = "tsv_limited"."TS_CODE"
      join "CWMS_20"."AT_TS_EXTENTS"
        on (
          "CWMS_20"."AT_TS_EXTENTS"."TS_CODE" = "tsv_limited"."TS_CODE"
          and "CWMS_20"."AT_TS_EXTENTS"."VERSION_TIME" = "tsv_limited"."VERSION_DATE"
        )
  )
select
  "max_values"."CWMS_TS_ID",
  "max_values"."DATE_TIME",
  "max_values"."VERSION_DATE",
  "max_values"."DATA_ENTRY_DATE",
  "max_values"."QUALITY_CODE",
  "max_values"."START_DATE",
  "max_values"."END_DATE",
  "CWMS_20"."CWMS_UTIL"."GET_DEFAULT_UNITS"(
    "CWMS_20"."CWMS_TS"."GET_BASE_PARAMETER_ID"("max_values"."TS_CODE"),
    'EN'
  ) "def_units",
  "max_values"."DATE_TIME" "max_date_time",
  "CWMS_20"."CWMS_UTIL"."CONVERT_UNITS"(
    "max_values"."VALUE",
    "max_values"."UNIT_ID",
    "CWMS_20"."CWMS_UTIL"."GET_DEFAULT_UNITS"(
      "CWMS_20"."CWMS_TS"."GET_BASE_PARAMETER_ID"("max_values"."TS_CODE"),
      'EN'
    )
  ) "value_at_max_date"
from "max_values"
where "max_values"."DATE_TIME" = "max_values"."max_date_time"

[rohaan-rma]

Old Query with over 50s runtime.

New and Improved query with under 1s runtime.

[rohaan-rma]

Integration Tests to show improved query time.

Old Query:

New Query:

[MikeNeilson]

Excellent Set that test up with a reasonable time assert, I think there's an existing check int the assertion. If not, and maybe there isn't since I wrote this... copy this over: https://github.com/opendcs/opendcs/blob/main/integrationtesting/fixtures/src/main/java/org/opendcs/fixtures/assertions/Waiting.java

You're local performance probably isn't the right target, but something like 5 seconds, just to make sure we aren't causing performance regressions later on.

[rohaan-rma]

A few changes:

• Only uses one table for tsv_limited if both start and end date are in the same year
• Updated category/group processing in the timeseries/recent endpoint to use optimized query

Old query:

New query:

Note: These integration tests are only to show time improvement from the old query to the new query. These tests are using the jdbc export and are not intended to be pushed anywhere. I am considering adding the .time(lessthan(5000 ms)) to the assertions in the current integration tests (Although the current IT aren't made for using with the export, so they would still pass under the time constraint because of a lack of populated data).

[rma-rripken]

Looks good to me

[rma-rripken]

This should probably use the constants you added.

[rohaan-rma]

Fixed. Sorry, I thought I had done that originally, but maybe after I resolved the merge conflicts this morning I might've accidentally changed them back.