Motivation
While exploring the codebase I noticed that several GIS preprocessing workflows are implemented independently across multiple modules.
These implementations produce the correct scientific results, but many of them follow nearly identical spatial-processing patterns with only small differences in column names or input datasets.
As the project continues to grow, this duplication may make future maintenance more difficult because improvements or bug fixes must be applied in several places.
This issue is not proposing any change to the scientific methodology or emission calculations.
Instead, it proposes discussing whether some common GIS preprocessing operations could be refactored into reusable utility functions while preserving the current scientific outputs.
Examples of repeated GIS workflows
During my review of the current upstream branch I found several repeated patterns.
1. Geographic coordinates → Domain CRS
Several modules independently perform the same workflow:
- Create Point geometries from longitude/latitude
- Assign EPSG:4326
- Reproject into the modelling domain CRS
Examples include:
- climate_trace
- npi
- oil_gas
- data parsers
A shared helper such as
"Calling it points_to_domain_crs() — basically wrapping the "lon/lat table → domain Lambert CRS GeoDataFrame" conversion in one function. No science changes, just DRYing out the CRS prep that's been copy-pasted in 4+ places."
could reduce repeated code while keeping behaviour identical.
2. ANZSIC prefix filtering
Several modules implement nearly identical logic for
- simplifying ANZSIC codes
- building prefix lists
- filtering with str.startswith()
The only differences are dataset-specific column names.
3. NPI facility deduplication
Both Oil & Gas and Waste perform essentially the same workflow:
- project into a metric CRS
- perform spatial nearest join
- remove facilities within a distance threshold
This could potentially be implemented as a reusable GIS utility.
4. Spatial joins for state well datasets
The state-specific Oil & Gas modules (NSW, QLD, NT, WA and Offshore) all follow a very similar pattern:
- spatial join between wells and titles
- determine activity start date
- filter by reporting period
- prepare a standard emission-source schema
Only dataset-specific field names differ.
5. Raster remapping
Several sectors perform the same pattern of
- remapping rasters to the modelling domain
- remapping again to the inventory domain
using similar code paths.
Possible direction
Rather than changing scientific algorithms, it may be helpful to gradually introduce reusable GIS utilities for common preprocessing tasks such as:
- CRS transformations
- point creation
- repeated spatial joins
- raster remapping wrappers
- spatial deduplication helpers
The goal would be to improve maintainability while preserving identical scientific behaviour.
Expected benefits
Potential benefits include:
- less duplicated GIS code
- easier maintenance
- more consistent spatial-processing behaviour
- simpler onboarding for new contributors
- easier future extension of new sectors
Discussion
Before spending time implementing anything, I wanted to open this issue first to ask whether this direction aligns with the project's design goals.
If the maintainers think this approach would be useful, I'd be happy to work on a focused pull request for one small GIS utility at a time.
Motivation
While exploring the codebase I noticed that several GIS preprocessing workflows are implemented independently across multiple modules.
These implementations produce the correct scientific results, but many of them follow nearly identical spatial-processing patterns with only small differences in column names or input datasets.
As the project continues to grow, this duplication may make future maintenance more difficult because improvements or bug fixes must be applied in several places.
This issue is not proposing any change to the scientific methodology or emission calculations.
Instead, it proposes discussing whether some common GIS preprocessing operations could be refactored into reusable utility functions while preserving the current scientific outputs.
Examples of repeated GIS workflows
During my review of the current upstream branch I found several repeated patterns.
1. Geographic coordinates → Domain CRS
Several modules independently perform the same workflow:
Examples include:
A shared helper such as
"Calling it points_to_domain_crs() — basically wrapping the "lon/lat table → domain Lambert CRS GeoDataFrame" conversion in one function. No science changes, just DRYing out the CRS prep that's been copy-pasted in 4+ places."
could reduce repeated code while keeping behaviour identical.
2. ANZSIC prefix filtering
Several modules implement nearly identical logic for
The only differences are dataset-specific column names.
3. NPI facility deduplication
Both Oil & Gas and Waste perform essentially the same workflow:
This could potentially be implemented as a reusable GIS utility.
4. Spatial joins for state well datasets
The state-specific Oil & Gas modules (NSW, QLD, NT, WA and Offshore) all follow a very similar pattern:
Only dataset-specific field names differ.
5. Raster remapping
Several sectors perform the same pattern of
using similar code paths.
Possible direction
Rather than changing scientific algorithms, it may be helpful to gradually introduce reusable GIS utilities for common preprocessing tasks such as:
The goal would be to improve maintainability while preserving identical scientific behaviour.
Expected benefits
Potential benefits include:
Discussion
Before spending time implementing anything, I wanted to open this issue first to ask whether this direction aligns with the project's design goals.
If the maintainers think this approach would be useful, I'd be happy to work on a focused pull request for one small GIS utility at a time.