Replace Climate TRACE oil and gas locations with petroleum wells datasets

changelog/161.feature.md

@@ -0,0 +1 @@
+Utilise state-based petroleum well and borehole datasets to locate oil and gas emission sources

docs/data-sources.md

@@ -193,7 +193,7 @@ Global dataset of CH4 flux estimates from termites.
 Termite emissions used in [Saunois et al. 2020](https://essd.copernicus.org/articles/12/1561/2020/)
 supplied by Simona Castaldi and Sergio Noce.
 
-- Dataset: [termite_emissions_2010-2016.nc][6]
+- Dataset: [termite_emissions_2010-2016.nc][5]
 - Resolution: 0.5 degree
 - Period: mean of 2010 – 2016
 - Updates: never
@@ -313,17 +313,78 @@ spatialised according to the [Land Use of Australia](#Land-Use-of-Australia) dat
 
 Oil and gas emissions reported in the
 [Australian UNFCCC Inventory](#Australian-UNFCCC-Inventory)
-are spatialised according to facility-level estimates.
+are spatialised according to locations of oil and gas boreholes/wells which lie
+within petroleum titles/leases that were active during the period of interest.
 
 ### Sources
 
-- Dataset: [Oil and gas production sources][5]
-    - Original source: [ClimateTrace](https://climatetrace.org/)
+- New South Wales datasets:
+  - [Data.NSW - Coal Seam Gas Boreholes](https://data.nsw.gov.au/data/dataset/coal-seam-gas-borehole)
+  - [Data.NSW - NSW Drillholes Petroleum](https://data.nsw.gov.au/data/dataset/nsw-drillholes-petroleum)
+  - [Data.NSW - NSW Exploration and Mining Titles](https://data.nsw.gov.au/data/dataset/nsw-mining-titles)
+- Northern Territory datasets:
+  - [STRIKE](http://strike.nt.gov.au/wss.html)
+    - Downloads -> Petroleum Titles and Pipeline Titles -> All Petroleum and Pipeline Titles Layers
+    - Downloads -> Drilling -> Petroleum Wells
+- Queensland datasets:
+  - [Queensland borehole series](https://www.data.qld.gov.au/dataset/queensland-borehole-series)
+  - [Queensland mining and exploration tenure series](https://www.data.qld.gov.au/dataset/queensland-mining-and-exploration-tenure-series)
+- South Australia datasets:
+  - [PEPS SA Downloads](https://peps.sa.gov.au/more/excels/)
+    - Well Details and Locations
+    - Monthly Production by Completion
+- Western Australia datasets:
+  - [WA Petroleum Wells (DMIRS-025)](https://catalogue.data.wa.gov.au/dataset/wa-onshore-petroleum-wells-dmirs-025)
+  - [WA Petroleum Titles (DMIRS-011)](https://catalogue.data.wa.gov.au/dataset/wa-petroleum-titles-dmirs-011)
+- Offshore datasets:
+  - [National Offshore Petroleum Information Management System (NOPIMS)](https://www.nopta.gov.au/maps-and-public-data/nopims-info.html)
+  - [National Electronic Approvals Tracking System](https://public.neats.nopta.gov.au/)
+
+Locations of every borehole/drillhole/well in the public datasets from NSW, NT,
+QLD, SA, WA and NOPTA are correlated with petroleum production titles and
+filtered to only bores involved in petroleum production where the title period
+overlaps with the prior period of interest.
+
+The national inventory total for oil and gas emissions is divided evenly between
+these sites. The listed point location for each site is mapped to the relevant
+domain grid cell, where emissions are allocated.
 
-The national inventory total for oil and gas emissions is pro-rated to the
-location of every facility noted in the ClimateTrace data which is
-listed for the chosen period. The listed point location for each
-climate trace emission is mapped to the relevant domain grid cell.
+### Considerations
+
+The approach used to spatialise the oil and gas sector has several known flaws.
+
+1. Capped wells
+
+First and foremost, although some datasets list many bores/wells as "capped" or
+"abandoned", they don't include the date when the capping occurred. For this
+reason we consider every production well to be an emission source until the
+date of expiry of the title. This could be incorrect in both ways: wells very
+likely stop emitting methane when they are capped, and in that case this leads
+to many false positives within the datasets. Alternatively, capped wells may
+continue to emit methane after production (and the title) has ended, leading to
+missing emissions on abandoned fields.
+
+2. Attribution of inventory emissions
+
+Attribution of equal emissions to every "active" bore/well is also
+very naive. Wells for different resources (i.e. oil vs coal seam gas) or in
+different regions (WA vs NSW) or in different infrastructure (onshore vs
+offshore) are likely to have very different emission profiles. Until we have
+solid evidence of what these profiles might be, we cannot model them.
+
+3. Refineries and pipelines
+
+Several types of major infrastructure are currently missing from this approach:
+refineries and pipelines. This is a major omission, as there is reasonable
+suspicion that the majority of emissions occur at processing facilities. We
+hope to add these facilities at a later date.
+
+4. Missing regions
+
+- Victoria
+  - oil and gas extraction currently entirely offshore, present in NOPTA dataset
+- ACT
+  - no oil or gas production to date
 
 
 ## Sector: Stationary
@@ -350,5 +411,4 @@ spatialised according to the [Land Use of Australia](#Land-Use-of-Australia) dat
 [2]: https://openmethane.s3.amazonaws.com/prior/inputs/EntericFermentation.nc
 [3]: https://openmethane.s3.amazonaws.com/prior/inputs/ch4-electricity.csv
 [4]: https://openmethane.s3.amazonaws.com/prior/inputs/landuse-sector-map.csv
-[5]: https://openmethane.s3.amazonaws.com/prior/inputs/oil-and-gas-production-and-transport_emissions-sources.csv
-[6]: https://openmethane.s3.amazonaws.com/prior/inputs/termite_emissions_2010-2016.nc
+[5]: https://openmethane.s3.amazonaws.com/prior/inputs/termite_emissions_2010-2016.nc

notebooks/oil_gas_locations.py

@@ -0,0 +1,40 @@
+import datetime
+import geopandas as gpd
+import pathlib
+from openmethane_prior.lib import PriorConfig
+from openmethane_prior.lib.data_manager.manager import DataManager
+from openmethane_prior.sectors.oil_gas.emission_sources.all_sources import all_emission_sources
+prior_config = PriorConfig(
+    domain_path="https://openmethane.s3.amazonaws.com/domains/aust10km/v1/domain.aust10km.nc",
+    inventory_domain_path="https://openmethane.s3.amazonaws.com/domains/aust10km/v1/domain.aust10km.nc",
+    start_date=datetime.datetime.fromisoformat("2022-12-01"),
+    input_path=pathlib.Path("../data/inputs"),
+    input_cache=pathlib.Path("../data/.cache"),
+)
+prior_config.prepare_paths()
+prior_config.load_cached_inputs()
+data_manager = DataManager(
+    data_path=prior_config.input_path,
+    prior_config=prior_config,
+)
+
+emission_sources_df = all_emission_sources(
+    data_manager=data_manager,
+    prior_config=prior_config,
+)
+
+au_shp = gpd.GeoDataFrame.from_file("~/Downloads/STE_2021_AUST_SHP_GDA2020/STE_2021_AUST_GDA2020.shx").to_crs("EPSG:4326")
+au_map = au_shp.plot(color='white', edgecolor='#CCCCCC', figsize=(32,24))
+region_extents = {
+    "QLD": (137, 155, -30, -20),
+    "VIC": (146, 150, -41, -36),
+    "WA": (110, 130, -35, -10),
+}
+region = "QLD" # set to "QLD", "VIC" or "WA" to render a sub-region
+if region is not None:
+    au_map.set_xlim(region_extents[region][0], region_extents[region][1])
+    au_map.set_ylim(region_extents[region][2], region_extents[region][3])
+emission_sources_df.to_crs("EPSG:4326").plot(ax=au_map, markersize=3, column="site_type", legend=True)
+au_map
+
+

pyproject.toml

@@ -18,14 +18,17 @@ dependencies = [
     "rioxarray>=0.15.5,<0.16",
     "pyproj>=3.6.1,<4",
     "pandas>=2.2.2,<3",
-    "geopandas>=0.14.4,<0.15",
+    "geopandas>=1.1.3,<2",
     "python-dotenv>=1.0.1,<2",
     "colorama>=0.4.6,<0.5",
     "cdsapi>=0.7.3,<0.8",
     "shapely>=2.0.4,<3",
     "environs>=11.0.0,<12",
     "prettyprinter>=0.18.0,<0.19",
     "pytest-mock>=3.15.1,<4",
+    "bmi-arcgis-restapi>=2.4.15",
+    "owslib>=0.35.0",
+    "openpyxl>=3.1.5",
 ]
 
 [dependency-groups]

src/openmethane_prior/lib/__init__.py

@@ -39,6 +39,7 @@
     load_zipped_pickle,
     redistribute_spatially,
     save_zipped_pickle,
+    rows_in_period,
 )
 
 import openmethane_prior.lib.logger as logger

src/openmethane_prior/lib/data_manager/parsers.py

@@ -15,10 +15,71 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+import geopandas as gpd
 import pandas as pd
+import pyproj
 
 from openmethane_prior.lib.data_manager.source import ConfiguredDataSource
 
 
 def parse_csv(data_source: ConfiguredDataSource) -> pd.DataFrame:
+    """Read and parse a ConfiguredDataSource CSV asset as a pandas DataFrame."""
     return pd.read_csv(data_source.asset_path)
+
+
+def parse_geo(data_source: ConfiguredDataSource, source_crs: pyproj.CRS = None):
+    """Read and parse a file containing a collection of geometry vector data
+    into a geopandas GeoDataFrame. Asset file type can be anything supported by
+    pyogrio, which includes GeoJSON, GeoPackage, Shapefiles, etc."""
+    geo_df = gpd.read_file(data_source.asset_path)
+
+    if geo_df.crs is None:
+        if source_crs is not None:
+            geo_df = geo_df.set_crs(source_crs)
+        else:
+            raise ValueError("parse_geo could not determine CRS, must be called manually with source_crs parameter")
+
+    # convert the geometries into the prior projection to ensure downstream
+    # comparisons are done using the same coordinate system
+    geo_df = geo_df.to_crs(data_source.prior_config.crs)
+
+    return geo_df
+
+
+def parse_xlsx(data_source: ConfiguredDataSource) -> pd.DataFrame:
+    """Read and parse a ConfiguredDataSource XLSX asset as a pandas DataFrame."""
+    return pd.read_excel(data_source.asset_path)
+
+
+def parse_geo_xlsx(
+    x_column: str,
+    y_column: str,
+    source_crs: pyproj.CRS | str,
+):
+    """Create a parser which will read an Excel-based ConfiguredDataSource,
+    extracting coordinates from columns in x_column and y_column, and project
+    coordinates from the source_crs to the PriorConfig domain CRS.
+
+    This can be used to configure a DataSource like:
+      DataSource(
+        file_path="example.xlsx",
+        parse=parse_geo_xlsx("x_col", "y_col", "EPSG:7844")
+      )
+    """
+    def _parser(data_source: ConfiguredDataSource):
+        df = parse_xlsx(data_source)
+
+        df["geometry"] = gpd.points_from_xy(
+            x=df[x_column],
+            y=df[y_column],
+            crs=source_crs,
+        )
+        geo_df = gpd.GeoDataFrame(df)
+
+        # convert the geometries into the prior projection to ensure downstream
+        # comparisons are done using the same coordinate system
+        geo_df = geo_df.to_crs(data_source.prior_config.crs)
+
+        return geo_df
+
+    return _parser

src/openmethane_prior/lib/utils.py

@@ -29,6 +29,8 @@
 import typing
 
 import numpy as np
+import pandas as pd
+import geopandas as gpd
 from numpy.typing import ArrayLike
 from urllib.parse import urlparse
 import xarray as xr
@@ -196,3 +198,23 @@ def is_url(maybe_url: str) -> bool:
     """
     parsed = urlparse(maybe_url)
     return parsed.scheme != "" and parsed.netloc != ""
+
+
+DF = typing.TypeVar("DF", bound=pd.DataFrame | gpd.GeoDataFrame)
+def rows_in_period(
+    df: DF,
+    start_date: datetime.date,
+    end_date: datetime.date,
+    start_field: str = "start_date",
+    end_field: str = "end_date",
+) -> DF:
+    """Return the rows of the DataFrame df where the period between df[start_field]
+    and df[end_field] overlaps with the period between start_date and end_date."""
+    midnight = datetime.time(0, 0)
+    period_start = datetime.datetime.combine(date=start_date, time=midnight)
+    period_end = datetime.datetime.combine(date=end_date + datetime.timedelta(days=1), time=midnight)
+
+    return df[
+        (df[start_field] <= np.datetime64(period_end))
+        & ((np.isnat(df[end_field])) | (df[end_field] >= np.datetime64(period_start)))
+    ]

src/openmethane_prior/sectors/oil_gas/README.md

@@ -0,0 +1,102 @@
+
+# Oil and Gas Sector
+
+The oil and gas sector is the most complicated sector implementation in the
+Australia prior. There's no pre-existing high quality dataset that can be
+used to both locate and scale emission sources from the sector, and no clear
+spatial proxy.
+
+Oil and gas extraction occurs in "fields" of wells or bores, somtimes located
+offshore. Extracted resources are transported through pipelines to various
+pieces of infrastructure to be compressed or combined, and eventually to
+refinery facilities. Methane can be lost to the atmosphere for various reasons
+at any of these locations, resulting in physically dispersed infrastructure
+which is not well documented by the private companies which operate it.
+
+## Methodology
+
+Emissions are likely to occur in three possible parts of the oil and gas
+infrastructure:
+- wells or boreholes where oil or gas are extracted
+- processing facilities
+- pipelines connecting wells and other infrastructure
+
+### Wells and boreholes
+
+#### Locations
+
+Oil and gas extraction are covered by petroleum mining laws in Australia, and
+most Australian states provide public datasets of petroleum mining titles
+(aka leases/licenses/tenements). Some states also provide public datasets
+of drill/bore/well locations.
+
+Drillhole datasets typically include enough detail to determine which bores
+or wells are used for petroleum production, giving us precise locations for
+emission sources.
+
+#### Activity period
+
+Drillhole datasets typically include the approximate date a bore was drilled
+(sometimes just the year), but don't include the date a bore was capped or
+depleted. Although some bore datasets do include the bore status, ie CAPPED,
+this is the status **at the time the dataset was last updated**, and not
+necessarily the status in the period of interest for the prior.
+
+We use a very rough method to determine whether a single bore may have been
+producing emissions during a target period (`start_date` to `end_date`).
+- if the bore drill date is later than `end_date`, no emissions are possible
+- correlate the bore coordinates with all petroleum production titles it
+  is contained by
+- if the range from `start_date` to `end_date` overlaps with the range from
+  title grant date to title expiry date, emissions are possible
+
+This method has obvious problems:
+
+1. emissions are unlikely to start immediately after drilling, however without
+   a more accurate start date, this is at least a lower bound
+2. most bores/wells will be depleted or capped well before the petroleum title
+   expires, however without an accurate capping date, this is at least an upper
+   bound.
+3. there is potentially a long period (years) where an entire field / title
+   has stopped producing, but the title is still active.
+
+On a coarse grid such as our 10x10km grid, 1. and 2. probably aren't serious,
+because emissions will be dispersed amongst a number of points within each grid
+cell.
+
+The biggest issue is 3., as it will place emissions in some fields, possibly
+for years after production has ceased. Assuming that capped wells don't
+continue to emit methane, this will misallocate potentially significant
+emissions. Unfortunately we currently don't have a better solution.
+
+#### South Australia
+
+Unlike other states, South Australia also provides monthly per-well production
+amounts for both oil and gas. For South Australia, only wells which produced
+either oil or gas within the period of interest will be considered an emission
+source.
+
+
+### Processing facilities
+
+Not yet implemented.
+
+### Pipelines
+
+Public datasets do exist which record the shape and status of oil and gas
+pipelines in Australia, however there are several issues using these shape
+files to estimate emissions.
+
+1. Lack of temporal data
+
+Pipeline datasets list the current status of pipelines, but don't necessarily
+include the dates pipelines actually began operating.
+
+2. Lack of leakage estimates
+
+We're not currently aware of any research or data on how much methane leaks
+from pipelines, or where leakage occurs. This makes it hard to attribute a
+specific volume of emission to specific locations along any given pipeline.
+
+For now, pipelines are not included in our bottom up estimate. If better data
+or research becomes available in the future, this would be a welcome addition.