Replace Climate TRACE oil and gas locations with petroleum wells datasets#161
Conversation
aethr
force-pushed
the
150/oil-gas-wells
branch
6 times, most recently
from
0f4372e to
9b16528
Compare
|
notebooks/README.md: I presume it's also possible to run the notebooks .../sectors/oil_gas/data/nopta_titles.py: There are hardcoded urls in .../sectors/oil_gas/data/nsw_drillholes.py: same as above .../sectors/oil_gas/data/nsw_titles.py: Is that NSW dataserver url now .../sectors/oil_gas/data/qld_boreholes.py: url again .../sectors/oil_gas/data/qld_leases.py and again .../sectors/oil_gas/data/sa_wells.py: this is how I expected the urls .../sectors/oil_gas/data/wa_titles.py: hardcoded url .../sectors/oil_gas/data/wa_wells.py: hardcoded url .../oil_gas/emission_sources/emission_source.py: The function name .../oil_gas/emission_sources/nsw_sources.py: this holds for all these |
I've tried to carry the jupyter torch (which was added by the Climate Resource folks) but I'm honestly not sure I'm getting much use out of it. I've never tried to run these scripts via cli because fundamentally they're being used to explore data visually with maps and charts. I think a better alternative to jupyter exists (marimo) which stores notebooks in .py by default, so I might experiment a bit here and see if I can make these more valuable. If not, I'm inclined to delete the whole folder.
My convention thus far has been to include a comment above every DataSource that includes a link to the origin of the data. The nopta_titles file includes this: For most DataSources which include a
I don't feel strongly about doing this or not doing it. I would probably suggest that if we want to centralise the URL, we might move both of these DataSources and their supporting logic into a single file?
When talking about data structures I would expect "normalise" to mean "make them all the same". The intent here was a function which:
This would allow us to combine multiple sets of emission sources and know we're dealing with a homogeneous structure. Do you have a suggestion for a word which would communicate that more clearly?
To clarify, are you suggesting that we rename import .offshore_sources
offshore_df = offshore_sources.emission_sources(
start_date=start_date,
end_date=end_date,
offshore_wells_da=offshore_wells_da,
offshore_titles_da=offshore_titles_da,
) |
|
Replying to several comments at once here, let me know if there's a
more github'ish way of doing this.
.../sectors/oil_gas/data/nopta_titles.py: There are hardcoded urls in
here. Perhaps leave a note for future maintainers that if this
breaks because of changed locations then you should look here
My convention thus far has been to include a comment above every DataSource that includes a
link to the origin of the data.
For most DataSources which include a url param directly, this is as close as possible. Is
your feedback that you'd like this to be closer to the hard-coded urls that are found in
custom fetch methods?
I preferred the way you handled it in:
src/openmethane_prior/sectors/oil_gas/data/nt_titles.py
where the url was added to the DataSource instance. That's a form used
a lot through the code now. If that's not sensible then I'd suggest
noting the presence of these urls in a developers-notes.md or similar file.
I don't feel strongly about doing this or not doing it. I would probably suggest that if we
want to centralise the URL, we might move both of these DataSources and their supporting
logic into a single file?
I think that's a good solution then define the variable at the module level.
When talking about data structures I would expect "normalise" to mean "make them all the
same". The intent here was a function which:
* ensures all emission sources include a minimum set of fields
* ensures all geometries use the same projection
This would allow us to combine multiple sets of emission sources and know we're dealing
with a homogeneous structure. Do you have a suggestion for a word which would communicate
that more clearly?
My concern is probably only one for mathematicians. When I map
disparate sources on to a common data structure I tend to call the
functions Canonicalise but I realise that has its own freight in the
IT world. I'd leave it.
.../oil_gas/emission_sources/nsw_sources.py: this holds for all these
regional files. Is it clearer to give the method a similar name and
just differentiate by the module? This might make it clearer they're
all doing versions of the same thing
To clarify, are you suggesting that we rename nsw_emission_sources et al. to just
emission_sources so that all regional implementations just implement a emission_sources
function? Certainly the intent in naming them all "*_emission_sources" was to make it clear
that they're all doing versions of the same thing. If we gave them all the exact same name,
what would the all_sources implementation look like? Something like:
I'd import the relevant modules, put them in a list and iterate over
them calling their .emission_sources method. Or define a class and
instantiate it for each jurisdiction. This looks to me like a
good analogy with how we do process_emissions.
|
The
That being the case, I would probably combine each of the regional sets of wells/titles into a single file for each region and apply the same pattern.
Standardise?
I don't think I agree in this case. There are three differences between this and
I think the last reason is probably the strongest in favour of leaving the emission_source methods with different names. They're similar, but take different arguments and return subtly different outputs. |
|
agree with maintaining emission_sources names as they are
|
aethr
force-pushed
the
150/oil-gas-wells
branch
from
d7714cb to
056b605
Compare
aethr
force-pushed
the
150/oil-gas-wells
branch
from
f5850fd to
05cfa61
Compare
aethr
force-pushed
the
150/oil-gas-wells
branch
from
78c997b to
a46e1e3
Compare
This reads the asset file using geopandas, and returns a GeoDataFrame, which is a pandas DataFrame that supports vector geometry fields.
When multiple GeoDataFrame objects are used in a layer and often combined, they must share a common CRS. The CRS of the domain projection may not be known at the time that parse_geo is run, so for now just normalise to a standard WGS84 projection.
Combine QLD boreholes dataset with petroleum leases dataset to extract producing wells with an activity period matching the lease period.
NOPTA is the National Offshore Petroleum Title Administrator who manages Australian offshore resource titles. Using their titles and wells datasets we can identify likely offshore locations where oil and gas are being extracted within a prior period.
Combine NSW drillholes dataset with petroleum leases dataset to extract producing wells with an activity period matching the period of interest.
Combine WA petroleum wells dataset with petroleum titles dataset to extract producing wells with an activity period matching the title period.
Emission sources outputs from each region are normalised into a single list which shares the emissions source structure. The offshore (NOPTA) dataset has significant overlap with WA and VIC datasets, with many wells occurring in both. In the case of VIC fields dataset, the NOPTA dataset has more detail and includes all the same sources, so the simplest solution is not to use the VIC data. In the case of WA, the WA dataset does include some wells that are not found in NOPTA, so we want to combine both datasets but remove any duplicates. The two datasets store coords with slightly different precision, so we allow for a small amount of distance between "duplicate" points.
Originally the idea behind normalise_emission_sources was to ensure each function returned a function which was already in the right format, or raise an error if that failed. However, from a functional standpoint it doesn't matter if sources are normalised until they're combined, which happens in the calling function. If we want to ensure each sources function returns the right columns, we can do this with a unit test instead of a runtime test, as we don't expect the format to change underneath us.
Since we have the config available to transform the coordinate system during parsing, we can avoid downstream complexity and errors by already using the same CRS as the rest of the data.
This replaces an outdated Climate TRACE dataset with only 18 approximate locations with the locations of all petroleum-related drillholes/boreholes/wells available in public datasets from Australian states. This approach has several drawbacks, first of all a high rate of false positives due to many of the wells being capped and abandoned. The lack of concrete data about when a well was capped leads us to continue allocating emissions to it during any period that falls within that well's petroleum title. A second weakness is the naive approach which allocates equal emissions to every borehole, which is certainly far from accurate. Even with these downsides, this approach will still likely result in more accurate dispersal of emissions from this sector than allocating all emissions to 18 points. Furthermore, many of these issues will be minimised by sourcing actual emissions per-facility or per-title from Safeguard Mechanism data sources, which will come in a future commit.
These datasets do update frequently, and we don't want tests to break arbitrarily based on updates to source data.
Combine Northern Territories petroleum wells dataset with petroleum titles dataset to generate a list of production wells within petroleum titles that are active within the period of interest.
Combine South Australia petroleum wells dataset with well production dataset to generate a list of production wells which recorded oil or gas production within the period of interest.
The original XLSX source for this file is 50MB+ and takes more than 60 seconds to parse. Since we only care about the data, and not the Excel functionality, saving as a CSV reduces parsing time to less than 5s. Although this won't make a difference in production where the source file is fetched and parsed either way, it significantly speeds up development and testing.
Every well and titles dataset for each state comes from the same source, and sometimes share other fetching or parsing functionality. This moves the two DataSource definitions for each regional source into a single file.
This doesn't result in a functional change, but has been requested in PR review as a more self-documenting way to keep the source URL closer to the DataSource which includes the documentation about where the URL was discovered.
The reference dataset was updated 22/04/2026, and appears to have cleaned up some of the `status` and other field values. Filters have been updated and test cases have been adjusted to remove field values that no longer appear in the data.
aethr
force-pushed
the
150/oil-gas-wells
branch
from
344c6ec to
cd1d00d
Compare
2 participants
Description
As part of incorporating Safeguard Mechanism methane estimates into the prior, we must first build up a list of locations which correspond to petroleum titles/leases where emissions have been reported.
Most Australian states publish a public dataset of boreholes/drillholes which include bores drilled for petroleum production. By locating petroleum bores that occur within active petroleum mining title/lease areas, we can build a dataset of likely emission points within a given time period. While this certainly contains many false positives due to lack of published data when wells are capped/depleted, it is already a much more comprehensive dataset than our previous implementation.
One clear disadvantage of this approach is in allocating an amount of emissions to each point. The CT dataset includes an emission estimate for each location, which we can scale to the sector total from the ANGA inventory. The currently approach doesn't have any proxy for emission quantity, meaning emissions must be distributed naively across the board, which is bound to be inaccurate. However, this should be rectified when we integrate SGM emissions which will cover the majority of emissions in the sector.
This PR also does not include processing facilities or pipelines, which will be important to include for an accurate picture of emissions.
This PR supersedes #152 , but does not currently attempt to implement any integration of Safeguard Mechanism emissions, which will come in a later PR.
Checklist
Please confirm that this pull request has done the following:
changelog/Notes
I ran an analysis comparing the emission sources derived in this PR with the previous Climate TRACE dataset. The CT dataset only contains 19 locations which roughly correspond to entire oil and gas fields. The new emissions sources dataset in this PR includes 9,946 geographic points corresponding to bores and wellheads within active petroleum titles.
Of the 19 existing locations, most have an equivalent point in the new dataset less than 1,000m away. The table below shows the distance from each point in the CT dataset to the closest point in the new approach.
The locations furthest away from an emission source in the new dataset are the Gippsland and Prelude, both representing offshore fields. After plotting the old locations on a map alongside points from the NOPTA (offshore) wells dataset, it appears that the CT points are possibly a "middle point" between many wells / fields in the area. However once the dataset is reduced to only "Development" wells, the CT points aren't necessarily that close to producing clusters.
Edit: updated analysis after adding NT well data.