TEST: Update glean expiration policies #827
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenWu
commented
Aug 21, 2024
There was a problem hiding this comment.
I went through the 102 tables to make sure the new values match what's in probe scraper and check if anything will be deleted with this change. Coincidentally, the firefox ios sync pings were created 400 days ago and the default expiration set for firefox ios is 400 days so these would start to get dropped starting tomorrow. We should probably increase the retention at least until the expiry alerts are ready.
All the other tables should be correct and nothing else will get immediately dropped with this change.
Query to get the first partition age of the 102 tables:
SELECT
*,
DATE_DIFF(run_date, first_partition_current, DAY) AS oldest_partition_age,
FROM
`moz-fx-data-shared-prod.monitoring_derived.table_partition_expirations_v1`
WHERE
run_date = "2024-08-20"
AND CONCAT(dataset_id, ".", table_id) IN (
"firefox_desktop_background_defaultagent_stable.baseline_v1",
"firefox_desktop_background_defaultagent_stable.events_v1",
"firefox_desktop_background_defaultagent_stable.metrics_v1",
"firefox_desktop_background_update_stable.bounce_tracking_protection_v1",
"firefox_desktop_background_update_stable.broken_site_report_v1",
"firefox_desktop_background_update_stable.fog_validation_v1",
"firefox_desktop_background_update_stable.use_counters_v1",
"firefox_desktop_stable.bounce_tracking_protection_v1",
"firefox_desktop_stable.broken_site_report_v1",
"firefox_desktop_stable.use_counters_v1",
"mlhackweek_search_stable.baseline_v1",
"mlhackweek_search_stable.events_v1",
"mlhackweek_search_stable.metrics_v1",
"org_mozilla_fenix_nightly_stable.bounce_tracking_protection_v1",
"org_mozilla_fenix_nightly_stable.broken_site_report_v1",
"org_mozilla_fenix_nightly_stable.crash_v1",
"org_mozilla_fenix_nightly_stable.fog_validation_v1",
"org_mozilla_fenix_nightly_stable.use_counters_v1",
"org_mozilla_fenix_stable.bounce_tracking_protection_v1",
"org_mozilla_fenix_stable.broken_site_report_v1",
"org_mozilla_fenix_stable.crash_v1",
"org_mozilla_fenix_stable.fog_validation_v1",
"org_mozilla_fenix_stable.use_counters_v1",
"org_mozilla_fennec_aurora_stable.bounce_tracking_protection_v1",
"org_mozilla_fennec_aurora_stable.broken_site_report_v1",
"org_mozilla_fennec_aurora_stable.crash_v1",
"org_mozilla_fennec_aurora_stable.fog_validation_v1",
"org_mozilla_fennec_aurora_stable.use_counters_v1",
"org_mozilla_firefox_beta_stable.bounce_tracking_protection_v1",
"org_mozilla_firefox_beta_stable.broken_site_report_v1",
"org_mozilla_firefox_beta_stable.crash_v1",
"org_mozilla_firefox_beta_stable.fog_validation_v1",
"org_mozilla_firefox_beta_stable.use_counters_v1",
"org_mozilla_firefox_stable.bounce_tracking_protection_v1",
"org_mozilla_firefox_stable.broken_site_report_v1",
"org_mozilla_firefox_stable.crash_v1",
"org_mozilla_firefox_stable.fog_validation_v1",
"org_mozilla_firefox_stable.use_counters_v1",
"org_mozilla_focus_beta_stable.baseline_v1",
"org_mozilla_focus_beta_stable.bounce_tracking_protection_v1",
"org_mozilla_focus_beta_stable.broken_site_report_v1",
"org_mozilla_focus_beta_stable.crash_v1",
"org_mozilla_focus_beta_stable.deletion_request_v1",
"org_mozilla_focus_beta_stable.events_v1",
"org_mozilla_focus_beta_stable.fog_validation_v1",
"org_mozilla_focus_beta_stable.metrics_v1",
"org_mozilla_focus_beta_stable.use_counters_v1",
"org_mozilla_focus_nightly_stable.baseline_v1",
"org_mozilla_focus_nightly_stable.bounce_tracking_protection_v1",
"org_mozilla_focus_nightly_stable.broken_site_report_v1",
"org_mozilla_focus_nightly_stable.crash_v1",
"org_mozilla_focus_nightly_stable.deletion_request_v1",
"org_mozilla_focus_nightly_stable.events_v1",
"org_mozilla_focus_nightly_stable.fog_validation_v1",
"org_mozilla_focus_nightly_stable.metrics_v1",
"org_mozilla_focus_nightly_stable.use_counters_v1",
"org_mozilla_focus_stable.baseline_v1",
"org_mozilla_focus_stable.bounce_tracking_protection_v1",
"org_mozilla_focus_stable.broken_site_report_v1",
"org_mozilla_focus_stable.crash_v1",
"org_mozilla_focus_stable.deletion_request_v1",
"org_mozilla_focus_stable.events_v1",
"org_mozilla_focus_stable.fog_validation_v1",
"org_mozilla_focus_stable.metrics_v1",
"org_mozilla_focus_stable.use_counters_v1",
"org_mozilla_ios_fennec_stable.addresses_sync_v1",
"org_mozilla_ios_fennec_stable.bookmarks_sync_v1",
"org_mozilla_ios_fennec_stable.creditcards_sync_v1",
"org_mozilla_ios_fennec_stable.history_sync_v1",
"org_mozilla_ios_fennec_stable.logins_sync_v1",
"org_mozilla_ios_fennec_stable.sync_v1",
"org_mozilla_ios_fennec_stable.tabs_sync_v1",
"org_mozilla_ios_firefox_stable.addresses_sync_v1",
"org_mozilla_ios_firefox_stable.bookmarks_sync_v1",
"org_mozilla_ios_firefox_stable.creditcards_sync_v1",
"org_mozilla_ios_firefox_stable.history_sync_v1",
"org_mozilla_ios_firefox_stable.logins_sync_v1",
"org_mozilla_ios_firefox_stable.sync_v1",
"org_mozilla_ios_firefox_stable.tabs_sync_v1",
"org_mozilla_ios_firefoxbeta_stable.addresses_sync_v1",
"org_mozilla_ios_firefoxbeta_stable.bookmarks_sync_v1",
"org_mozilla_ios_firefoxbeta_stable.creditcards_sync_v1",
"org_mozilla_ios_firefoxbeta_stable.history_sync_v1",
"org_mozilla_ios_firefoxbeta_stable.logins_sync_v1",
"org_mozilla_ios_firefoxbeta_stable.sync_v1",
"org_mozilla_ios_firefoxbeta_stable.tabs_sync_v1",
"org_mozilla_klar_stable.baseline_v1",
"org_mozilla_klar_stable.bounce_tracking_protection_v1",
"org_mozilla_klar_stable.broken_site_report_v1",
"org_mozilla_klar_stable.crash_v1",
"org_mozilla_klar_stable.deletion_request_v1",
"org_mozilla_klar_stable.events_v1",
"org_mozilla_klar_stable.fog_validation_v1",
"org_mozilla_klar_stable.metrics_v1",
"org_mozilla_klar_stable.use_counters_v1",
"org_mozilla_reference_browser_stable.crash_v1",
"pine_stable.bounce_tracking_protection_v1",
"pine_stable.broken_site_report_v1",
"pine_stable.deletion_request_v1",
"pine_stable.pageload_v1",
"pine_stable.use_counters_v1",
"treeherder_stable.events_v1"
)
ORDER BY
dataset_id,
table_idThere was a problem hiding this comment.
Current oldest partition is 399 days so data will be dropped if this get updated. The tables was created on 2023-07-17 (400 days ago) so it's just a coincidence that today would be the first day it start dropping data.
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
There was a problem hiding this comment.
oldest partition is 399 days
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
test diff for mozilla/mozilla-schema-generator#280