[FEATURE] Update observed mortality rate calculation

[GregorSchroeder]

Is your feature request related to a problem? Please describe

The current methodology used to calculate mortality rates for increment years does not align with the findings of SANDAG's preliminary mortality rate prediction project. The current methodology is described in the Wikipedia. Do not alter the jump-off year functionality of holding the last observed rates constant for horizon years. That work will be performed in #6.

Describe the solution you'd like

https://github.com/SANDAG/MortalityPredictionProject
Note that we may need to mix and match from old and new CDC WONDER products and potentially implement a bridging methodology. I leave this to @KPHSANDAG to describe and work through in subsequent comments.

Additional context

https://github.com/SANDAG/Cohort-Component-Model/blob/main/python/input_modules/death_rates.py
https://github.com/SANDAG/Cohort-Component-Model/wiki/Death-Rates

[KPHSANDAG]

The first step is to re-evaluate and correct the required datasets. The current CCM datasets use a varied collection of geographies consisting of five counties, California, multiple states, and national. For the larger purpose of forecasting for the San Diego Region, a better selection would be to primarily use local county data, and when not available shift to using state and national data. Details on this selection can be found later in this blurb.

The data downloaded from the CDC should be stratified by geography, single-year ages, sex, Hispanic origin, and race. We focus on the five races listed in the 1999-2020 CDC product, following the 1977 OMB standards: American Indian or Alaska Native, Asian or Pacific Islander, Black or African American, and White. Because Hispanic Origin lists either Hispanic or Latino, Not Hispanic or Latino, or Not Stated, we opt to include Hispanic as another race, where all races with the Hispanic or Latino Hispanic origin will be categorized as Hispanic. Conversely, all races that are not Hispanic are Not Hispanic or Latino Hispanic Origin, and Not Stated Hispanic Origins will be set aside for a later transformation.

To create a consistent time series, it is inevitable that both products of the CDC WONDER (1990-2020 and 2018-2023) will need to be utilized. The issue arises due to inconsistent race categories (i.e. the introduction of the “More than one race” category in the 2018-2023 CDC product). To combine the two products, we need a bridging methodology that will reassign a portion of the “More than one race” category to each of the other six race/ethnicities. This can be created following Table 3 of the Draft Provisional Guidance on the Implementation of the 1997 Standards for Federal Data on Race and Ethnicity. Table 3 of that document provides distribution breakdowns, which can be used to create fractions that serve as bridging factors for the More than one race category into the old four-category system (Tabulation Working Group 1999:161). Additionally, the CDC product does not provide population counts when stratified by county and single-year ages so population would have to be taken from the P-3: Complete State and County Projections Dataset from the State of California Department of Finance. The P-3, however, does include a Multiracial (MR) category, which does not have a direct mapping to the four-category race system used in the 1999-2020 CDC product, and so, the same bridging methodology would also need to be applied.

Afterwards, we apply transformations to the data to deal with suppressed and not stated deaths. Given that both CDC products contain numerous instances of suppressed values when stratified by County and Single-Year Ages, we opt to take five year moving averages. This means that when downloading data for the year 2003, we will be selecting the four preceding years in addition to 2003 (ex. 1999, 2000, 2001, 2002, 2003). This clusters the data to avoid some of the suppression.

Another issue in the data is the amount of “Not Stated”. The rows with "Not Stated" have associated death counts but no population counts, making them unattributable to a specific demographic. To address this, we inflate the attributable deaths by this proportion and recalculate the rates based on the inflated death counts.

The next step is to combine data from the three geographies corresponding to the following substitution methodology: 1) County rate, 2) State rate, 3) National rate, 4) Average rate (The average of the county rates from ages before and after the target age), 5) Previous rate (The rate of the previous age from the target age in the county dataset), 6) Next rate (The rate of the next age from the target age in the county dataset), 7) Imputation (This means that for deaths that are null, we follow a specific methodology to impute the deaths based on the population; while assuring we have no unrealistic deaths (i.e. deaths of >0 when population =0)). This methodology guarantees that we no rates are left null, and there will always be rates to use.

After combining the geographies, there will still be noise across the dataset. To address this, a smoothing function is introduced to smooth the data and prepare it for modelling. The parameters for the smoothing function will be included in the future modelling pipeline to determine the most optimal value(s).