Refactor input files

[aethr]

The problem

The prior utilises many input files as data sources for estimating emissions across different sectors. Inputs must be fetched, and sometimes pre-processed before they can be used. As the number of inputs grows, the current practice of keeping input filenames in environment variables and fetching everything at once will become inefficient.

Some inputs are utilised by many sectors, and others are only used once. Some must be preprocessed before they can be used, which is computationally expensive. Currently, sectors which all utilise the same input must be computed in the same method to reduce expensive re-processing. If we want to achieve #91, we must have a way to graph the dependencies between sectors and their inputs, so only necessary inputs can be fetched and processed.

Proposed solution

Create a "DataSource" concept in the codebase, which encapsulates each input file/source. At a minimum, a data source should include:

• a unique name
• a way for the file to be fetched (a URL or a method)
• whether the file is static (never changes) or dynamic (based on inputs such as time period)
• a method for preprocessing if necessary (ie parsing or re-gridding), which is cacheable

Each sector then includes a list of DataSources it needs, which can be coordinated by the top-level processes by:

• removing duplicates (ie multiple layers use the same DataSource)
• fetching inputs if necessary
• pre-processing if necessary

The results can then be provided back to each layer for use preparing emissions.

Definition of "done"

• Each input file in LayerInputs is converted to a DataSource object
• GFAS fetching is converted to a DataSource object
• Each sector module exports a data_sources list which includes all necessary inputs
• omPrior fetches any missing inputs before sector processing
• Each sector module uses data provided by the DataSources it specified

Additional context