The motivation
In theory, the architecture of openmethane-prior allows the user to specify any start_date and end_date, and the resulting output will include daily time steps with generated data for each day. This is easy to achieve in sector implementations where the data source is static (estimates for each day will be identical) or where the source data is provided daily and must simply be regridded (ie, GFAS).
In reality, many of the primary data sources provide annual data. For international/research data sources this annual data aligns with a calendar year, and adding support for a period of interest that crosses the calendar year boundary adds a great deal of complexity. The simplest solution is to throw an error in the sector implementation when it detects this scenario.
Similarly, Australian annual data sources typically align with the Australian financial year, offset by 6 months from the calendar year. The same conditions apply here, but sectors will throw an error if the start_date/end_date occur fall in different financial years.
This puts an effective limit on the start_date/end_date, making the maximum supported period 6 months, from YYYY-01-01 to YYYY-06-30 or from YYYY-07-01 to YYYY-12-31. In the current state, each data source must implement its own date checking and error logic to prevent these configurations.
The proposed solution
Since the output of the prior is a standardised format with daily data in a regular grid, you can get results over long periods using several approaches, which will all generate the exact same outcome:
- run the prior with start_date/end_date at the bounds of your desired period
- this results in a single output file with daily timesteps
- run the prior repeatedly for 1-day periods covering your total desired period
- this results an output file for each day, with the same format and structure
- run the prior repeatedly over n-day periods (weekly or monthly) covering your total desired period
- this results in an output file for each sub-period, with the same format and structure
If a single output file is desired, the latter approaches are still feasible, as NetCDF files are very simple to combine when they share a grid and format.
I propose two changes to simplify the existing situation:
- Add validation in PriorConfig to prevent
start_date and end_date in separate months
This neatly solves the annual alignment issue for calendar and financial years, because all dates within any calendar month are guaranteed to be in both the same financial year and calendar year. It allows sector implementations to be much simpler, as they have to do less validation, and can simply find the right data source for the month being requested.
- Add a "multi-month" prior script
For users who desire a single-file output covering a longer time period, this script would:
- break their desired date span into periods by month
- run the prior across each period, sharing a data folder
- combine the resulting outputs into a single file which includes a daily timestep for each day in the entire period
Alternatives
Some users may only desire to run a subset of layers, and these layers may already support the annual period desired. In this case, adding global limitations and extra scripts would complicate their workflow for little benefit. Leaving it up to each sector to define their own date validation and handling may be a better long term approach.
Not adding artificial limitations probably also makes modules within the prior more flexible for alternate purposes. For example, DataSources are likely to be useful outside of the prior, but do rely on a PriorConfig to function. This limitation would discourage DataSources to pursue date validation as it applies to their own implementation, making them less useful outside of prior runs.
Additional context
The motivation
In theory, the architecture of openmethane-prior allows the user to specify any
start_dateandend_date, and the resulting output will include daily time steps with generated data for each day. This is easy to achieve in sector implementations where the data source is static (estimates for each day will be identical) or where the source data is provided daily and must simply be regridded (ie, GFAS).In reality, many of the primary data sources provide annual data. For international/research data sources this annual data aligns with a calendar year, and adding support for a period of interest that crosses the calendar year boundary adds a great deal of complexity. The simplest solution is to throw an error in the sector implementation when it detects this scenario.
Similarly, Australian annual data sources typically align with the Australian financial year, offset by 6 months from the calendar year. The same conditions apply here, but sectors will throw an error if the start_date/end_date occur fall in different financial years.
This puts an effective limit on the start_date/end_date, making the maximum supported period 6 months, from YYYY-01-01 to YYYY-06-30 or from YYYY-07-01 to YYYY-12-31. In the current state, each data source must implement its own date checking and error logic to prevent these configurations.
The proposed solution
Since the output of the prior is a standardised format with daily data in a regular grid, you can get results over long periods using several approaches, which will all generate the exact same outcome:
If a single output file is desired, the latter approaches are still feasible, as NetCDF files are very simple to combine when they share a grid and format.
I propose two changes to simplify the existing situation:
start_dateandend_datein separate monthsThis neatly solves the annual alignment issue for calendar and financial years, because all dates within any calendar month are guaranteed to be in both the same financial year and calendar year. It allows sector implementations to be much simpler, as they have to do less validation, and can simply find the right data source for the month being requested.
For users who desire a single-file output covering a longer time period, this script would:
Alternatives
Some users may only desire to run a subset of layers, and these layers may already support the annual period desired. In this case, adding global limitations and extra scripts would complicate their workflow for little benefit. Leaving it up to each sector to define their own date validation and handling may be a better long term approach.
Not adding artificial limitations probably also makes modules within the prior more flexible for alternate purposes. For example, DataSources are likely to be useful outside of the prior, but do rely on a PriorConfig to function. This limitation would discourage DataSources to pursue date validation as it applies to their own implementation, making them less useful outside of prior runs.
Additional context