Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read.abares - Provides simple downloading, parsing and importing of Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES) data sources #667

Open
13 of 29 tasks
adamhsparks opened this issue Nov 2, 2024 · 85 comments

Comments

@adamhsparks
Copy link
Member

adamhsparks commented Nov 2, 2024

Submitting Author Name: Adam Sparks
Submitting Author Github Handle: @adamhsparks
Repository: https://github.com/adamhsparks/read.abares
Version submitted: 0.1.0
Submission type: Standard
Editor: @obrl-soil
Reviewers: TBD

Archive: TBD
Version accepted: TBD
Language: en


  • Paste the full DESCRIPTION file inside a code block below:
Type: Package
Package: read.abares
Title: Provides simple downloading, parsing and importing of Australian
    Bureau of Agricultural and Resource Economics and Sciences (ABARES)
    data sources
Version: 0.1.0
Authors@R: 
    person("Adam H.", "Sparks", , "[email protected]", role = c("cre", "aut"),
           comment = c(ORCID = "0000-0002-0061-8359"))
Description: Download and import data from the Australian Bureau of
    Agricultural and Resource Economics and Sciences (ABARES)
    <https://www.agriculture.gov.au/abares>.
License: MIT + file LICENSE
URL: https://github.com/adamhsparks/read.abares,
    https://adamhsparks.github.io/read.abares/
BugReports: https://github.com/adamhsparks/read.abares/issues
Imports: 
    cli,
    curl,
    data.table,
    lubridate,
    openxlsx2,
    purrr,
    readtext,
    sf,
    stars,
    stringr,
    terra,
    tidync,
    withr
Suggests: 
    knitr,
    pander,
    rmarkdown,
    roxyglobals,
    spelling,
    testthat (>= 3.0.0)
VignetteBuilder: 
    knitr
Config/roxyglobals/filename: globals.R
Config/roxyglobals/unique: FALSE
Config/testthat/edition: 3
Encoding: UTF-8
Language: en-US
LazyData: true
Roxygen: list(markdown = TRUE, roclets = c("collate", "namespace", "rd",
    "roxyglobals::global_roclet"))
RoxygenNote: 7.3.2

Scope

  • Please indicate which category or categories from our package fit policies this package falls under: (Please check an appropriate box below. If you are unsure, we suggest you make a pre-submission inquiry.):

    • data retrieval
    • data extraction
    • data munging
    • data deposition
    • data validation and testing
    • workflow automation
    • version control
    • citation management and bibliometrics
    • scientific software wrappers
    • field and lab reproducibility tools
    • database software bindings
    • geospatial data
    • text analysis
  • Explain how and why the package falls under these categories (briefly, 1-2 sentences):

Provides workflow assistance for fetching freely available data from the Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES), the science and economics research division of the Department of Agriculture, Fisheries and Forestry. Data include trade information, soil depth, agricultural survey data on production and model outputs of agricultural production useful for agricultural researchers.

  • Who is the target audience and what are scientific applications of this package?

    • Ag economists
    • Biometricians working in agriculture
    • Anyone else who is interested in Australian agricultural and economic data
  • Are there other R packages that accomplish the same thing? If so, how does yours differ or meet our criteria for best-in-category?

None that I'm aware of.

NA

  • If you made a pre-submission inquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

  • Explain reasons for any pkgcheck items which your package is unable to pass.

Technical checks

Confirm each of the following by checking the box.

This package:

Publication options

  • Do you intend for this package to go on CRAN?

  • Do you intend for this package to go on Bioconductor?

  • Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:

MEE Options
  • The package is novel and will be of interest to the broad readership of the journal.
  • The manuscript describing the package is no longer than 3000 words.
  • You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
  • (Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
  • (Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
  • (Please do not submit your package separately to Methods in Ecology and Evolution)

Code of conduct

@ropensci-review-bot
Copy link
Collaborator

Thanks for submitting to rOpenSci, our editors and @ropensci-review-bot will reply soon. Type @ropensci-review-bot help for help.

@ropensci-review-bot
Copy link
Collaborator

🚀

Editor check started

👋

@ropensci-review-bot
Copy link
Collaborator

Checks for read.abares (v0.1.0)

git hash: cd3d61e5

  • ✔️ Package name is available
  • ✔️ has a 'codemeta.json' file.
  • ✔️ has a 'contributing' file.
  • ✔️ uses 'roxygen2'.
  • ✔️ 'DESCRIPTION' has a URL field.
  • ✔️ 'DESCRIPTION' has a BugReports field.
  • ✔️ Package has at least one HTML vignette
  • ✔️ All functions have examples.
  • ✔️ Package has continuous integration checks.
  • ✖️ Package coverage failed
  • ✖️ R CMD check found 1 error.
  • ✔️ R CMD check found no warnings.
  • 👀 Function names are duplicated in other packages

Important: All failing checks above must be addressed prior to proceeding

(Checks marked with 👀 may be optionally addressed.)

Package License: MIT + file LICENSE


1. Package Dependencies

Details of Package Dependency Usage (click to open)

The table below tallies all function calls to all packages ('ncalls'), both internal (r-base + recommended, along with the package itself), and external (imported and suggested packages). 'NA' values indicate packages to which no identified calls to R functions could be found. Note that these results are generated by an automated code-tagging system which may not be entirely accurate.

type package ncalls
internal base 77
internal read.abares 10
internal methods 3
internal stats 3
internal tools 2
internal utils 2
imports data.table 21
imports purrr 6
imports httr2 5
imports terra 4
imports openxlsx2 3
imports sf 3
imports stars 3
imports tidync 3
imports stringr 2
imports readtext 1
imports cli NA
imports lubridate NA
imports withr NA
suggests knitr NA
suggests pander NA
suggests rmarkdown NA
suggests roxyglobals NA
suggests spelling NA
suggests testthat NA
linking_to NA NA

Click below for tallies of functions used in each package. Locations of each call within this package may be generated locally by running 's <- pkgstats::pkgstats(<path/to/repo>)', and examining the 'external_calls' table.

base

file.path (32), tempdir (13), c (12), dirname (7), url (4), lapply (2), list (2), list.files (2), append (1), message (1), nchar (1)

data.table

fread (9), fifelse (7), as.data.table (3), rbindlist (1), setcolorder (1)

read.abares

clear_cache (1), get_aagis_regions (1), get_abares_trade (1), get_abares_trade_regions (1), get_agfd (1), get_historical_forecast_database (1), get_historical_national_estimates (1), get_soil_thickness (1), inspect_cache (1), print.read.abares.agfd.nc.files (1)

purrr

map (4), modify_depth (1), quietly (1)

httr2

req_options (2), request (2), req_retry (1)

terra

rast (4)

methods

new (3)

openxlsx2

read_xlsx (3)

sf

read_sf (2), st_read (1)

stars

read_ncdf (2), read_stars (1)

stats

var (2), dt (1)

tidync

tidync (2), hyper_tibble (1)

stringr

str_locate (1), str_sub (1)

tools

R_user_dir (2)

utils

unzip (2)

readtext

readtext (1)

NOTE: Some imported packages appear to have no associated function calls; please ensure with author that these 'Imports' are listed appropriately.


2. Statistical Properties

This package features some noteworthy statistical properties which may need to be clarified by a handling editor prior to progressing.

Details of statistical properties (click to open)

The package has:

  • code in R (100% in 23 files) and
  • 1 authors
  • 1 vignette
  • no internal data file
  • 13 imported packages
  • 26 exported functions (median 9 lines of code)
  • 49 non-exported functions in R (median 15 lines of code)

Statistical properties of package structure as distributional percentiles in relation to all current CRAN packages
The following terminology is used:

  • loc = "Lines of Code"
  • fn = "function"
  • exp/not_exp = exported / not exported

All parameters are explained as tooltips in the locally-rendered HTML version of this report generated by the checks_to_markdown() function

The final measure (fn_call_network_size) is the total number of calls between functions (in R), or more abstract relationships between code objects in other languages. Values are flagged as "noteworthy" when they lie in the upper or lower 5th percentile.

measure value percentile noteworthy
files_R 23 83.4
files_vignettes 2 81.7
files_tests 22 95.6
loc_R 607 51.6
loc_vignettes 265 56.7
loc_tests 853 81.1
num_vignettes 1 58.8
n_fns_r 75 67.2
n_fns_r_exported 26 73.8
n_fns_r_not_exported 49 65.1
n_fns_per_file_r 2 34.9
num_params_per_fn 1 1.8 TRUE
loc_per_fn_r 11 33.0
loc_per_fn_r_exp 9 19.8
loc_per_fn_r_not_exp 15 49.9
rel_whitespace_R 21 56.7
rel_whitespace_vignettes 50 70.4
rel_whitespace_tests 10 66.3
doclines_per_fn_exp 25 23.7
doclines_per_fn_not_exp 0 0.0 TRUE
fn_call_network_size 36 57.9

2a. Network visualisation

Click to see the interactive network visualisation of calls between objects in package


3. goodpractice and other checks

Details of goodpractice checks (click to open)

3a. Continuous Integration Badges

R-CMD-check.yaml

GitHub Workflow Results

id name conclusion sha run_number date
11649832899 pages build and deployment success 980cc6 19 2024-11-03
11649818126 pkgdown.yaml success cd3d61 19 2024-11-03
11649818121 R-CMD-check.yaml failure cd3d61 19 2024-11-03
11649818125 test-coverage.yaml failure cd3d61 19 2024-11-03

3b. goodpractice results

R CMD check with rcmdcheck

R CMD check generated the following error:

  1. checking tests ...
    Running ‘spelling.R’
    Comparing ‘spelling.Rout’ to ‘spelling.Rout.save’ ... OK
    Running ‘testthat.R’
    ERROR
    Running the tests in ‘tests/testthat.R’ failed.
    Last 13 lines of output:
    Ran 1/1 deferred expressions
    There do not appear to be any files cached for {read.abares}.
    There do not appear to be any files cached for {read.abares}.

-- Locally Available {read.abares} Cached Files --------------------------------

  • test.R

-- Locally Available {read.abares} Cached Files --------------------------------

  • test.R
    Ran 1/1 deferred expressions
    Will return stars object with 612226 cells.
    No projection information found in nc file.
    Coordinate variable units found to be degrees,
    assuming WGS84 Lat/Lon.
    Killed

R CMD check generated the following test_fail:

  1. This file is part of the standard setup for testthat.

It is recommended that you do not modify it.

Where should you do additional test configuration?

Learn more about the roles of various files in:

* https://r-pkgs.org/testing-design.html#sec-tests-files-overview

* https://testthat.r-lib.org/articles/special-files.html

library(testthat)
library(read.abares)

Attaching package: 'read.abares'

The following object is masked from 'package:graphics':

plot

The following object is masked from 'package:base':

plot

test_check("read.abares")
Ran 1/1 deferred expressions
Ran 2/2 deferred expressions
Ran 1/1 deferred expressions
Ran 1/1 deferred expressions

-- Locally Available ABARES AGFD NetCDF Files ----------------------------------

  • f2022.c1991.p2022.t2022.nc
  • f2022.c1992.p2022.t2022.nc
  • f2022.c1993.p2022.t2022.nc
  • f2022.c1994.p2022.t2022.nc
  • f2022.c1995.p2022.t2022.nc
  • f2022.c1996.p2022.t2022.nc
  • f2022.c1997.p2022.t2022.nc
  • f2022.c1998.p2022.t2022.nc
  • f2022.c1999.p2022.t2022.nc
  • f2022.c2000.p2022.t2022.nc
  • f2022.c2001.p2022.t2022.nc
  • f2022.c2002.p2022.t2022.nc
  • f2022.c2003.p2022.t2022.nc
  • f2022.c2004.p2022.t2022.nc
  • f2022.c2005.p2022.t2022.nc
  • f2022.c2006.p2022.t2022.nc
  • f2022.c2007.p2022.t2022.nc
  • f2022.c2008.p2022.t2022.nc
  • f2022.c2009.p2022.t2022.nc
  • f2022.c2010.p2022.t2022.nc
  • f2022.c2011.p2022.t2022.nc
  • f2022.c2012.p2022.t2022.nc
  • f2022.c2013.p2022.t2022.nc
  • f2022.c2014.p2022.t2022.nc
  • f2022.c2015.p2022.t2022.nc
  • f2022.c2016.p2022.t2022.nc
  • f2022.c2017.p2022.t2022.nc
  • f2022.c2018.p2022.t2022.nc
  • f2022.c2019.p2022.t2022.nc
  • f2022.c2020.p2022.t2022.nc
  • f2022.c2021.p2022.t2022.nc
  • f2022.c2022.p2022.t2022.nc
  • f2022.c2023.p2022.t2022.nc

-- Locally Available ABARES AGFD NetCDF Files ----------------------------------

  • f2022.c1991.p2022.t2022.nc
  • f2022.c1992.p2022.t2022.nc
  • f2022.c1993.p2022.t2022.nc
  • f2022.c1994.p2022.t2022.nc
  • f2022.c1995.p2022.t2022.nc
  • f2022.c1996.p2022.t2022.nc
  • f2022.c1997.p2022.t2022.nc
  • f2022.c1998.p2022.t2022.nc
  • f2022.c1999.p2022.t2022.nc
  • f2022.c2000.p2022.t2022.nc
  • f2022.c2001.p2022.t2022.nc
  • f2022.c2002.p2022.t2022.nc
  • f2022.c2003.p2022.t2022.nc
  • f2022.c2004.p2022.t2022.nc
  • f2022.c2005.p2022.t2022.nc
  • f2022.c2006.p2022.t2022.nc
  • f2022.c2007.p2022.t2022.nc
  • f2022.c2008.p2022.t2022.nc
  • f2022.c2009.p2022.t2022.nc
  • f2022.c2010.p2022.t2022.nc
  • f2022.c2011.p2022.t2022.nc
  • f2022.c2012.p2022.t2022.nc
  • f2022.c2013.p2022.t2022.nc
  • f2022.c2014.p2022.t2022.nc
  • f2022.c2015.p2022.t2022.nc
  • f2022.c2016.p2022.t2022.nc
  • f2022.c2017.p2022.t2022.nc
  • f2022.c2018.p2022.t2022.nc
  • f2022.c2019.p2022.t2022.nc
  • f2022.c2020.p2022.t2022.nc
  • f2022.c2021.p2022.t2022.nc
  • f2022.c2022.p2022.t2022.nc
  • f2022.c2023.p2022.t2022.nc

-- Locally Available ABARES AGFD NetCDF Files ----------------------------------

  • f2022.c1991.p2022.t2022.nc
  • f2022.c1992.p2022.t2022.nc
  • f2022.c1993.p2022.t2022.nc
  • f2022.c1994.p2022.t2022.nc
  • f2022.c1995.p2022.t2022.nc
  • f2022.c1996.p2022.t2022.nc
  • f2022.c1997.p2022.t2022.nc
  • f2022.c1998.p2022.t2022.nc
  • f2022.c1999.p2022.t2022.nc
  • f2022.c2000.p2022.t2022.nc
  • f2022.c2001.p2022.t2022.nc
  • f2022.c2002.p2022.t2022.nc
  • f2022.c2003.p2022.t2022.nc
  • f2022.c2004.p2022.t2022.nc
  • f2022.c2005.p2022.t2022.nc
  • f2022.c2006.p2022.t2022.nc
  • f2022.c2007.p2022.t2022.nc
  • f2022.c2008.p2022.t2022.nc
  • f2022.c2009.p2022.t2022.nc
  • f2022.c2010.p2022.t2022.nc
  • f2022.c2011.p2022.t2022.nc
  • f2022.c2012.p2022.t2022.nc
  • f2022.c2013.p2022.t2022.nc
  • f2022.c2014.p2022.t2022.nc
  • f2022.c2015.p2022.t2022.nc
  • f2022.c2016.p2022.t2022.nc
  • f2022.c2017.p2022.t2022.nc
  • f2022.c2018.p2022.t2022.nc
  • f2022.c2019.p2022.t2022.nc
  • f2022.c2020.p2022.t2022.nc
  • f2022.c2021.p2022.t2022.nc
  • f2022.c2022.p2022.t2022.nc
  • f2022.c2023.p2022.t2022.nc

-- Locally Available ABARES AGFD NetCDF Files ----------------------------------

  • f2022.c1991.p2022.t2022.nc
  • f2022.c1992.p2022.t2022.nc
  • f2022.c1993.p2022.t2022.nc
  • f2022.c1994.p2022.t2022.nc
  • f2022.c1995.p2022.t2022.nc
  • f2022.c1996.p2022.t2022.nc
  • f2022.c1997.p2022.t2022.nc
  • f2022.c1998.p2022.t2022.nc
  • f2022.c1999.p2022.t2022.nc
  • f2022.c2000.p2022.t2022.nc
  • f2022.c2001.p2022.t2022.nc
  • f2022.c2002.p2022.t2022.nc
  • f2022.c2003.p2022.t2022.nc
  • f2022.c2004.p2022.t2022.nc
  • f2022.c2005.p2022.t2022.nc
  • f2022.c2006.p2022.t2022.nc
  • f2022.c2007.p2022.t2022.nc
  • f2022.c2008.p2022.t2022.nc
  • f2022.c2009.p2022.t2022.nc
  • f2022.c2010.p2022.t2022.nc
  • f2022.c2011.p2022.t2022.nc
  • f2022.c2012.p2022.t2022.nc
  • f2022.c2013.p2022.t2022.nc
  • f2022.c2014.p2022.t2022.nc
  • f2022.c2015.p2022.t2022.nc
  • f2022.c2016.p2022.t2022.nc
  • f2022.c2017.p2022.t2022.nc
  • f2022.c2018.p2022.t2022.nc
  • f2022.c2019.p2022.t2022.nc
  • f2022.c2020.p2022.t2022.nc
  • f2022.c2021.p2022.t2022.nc
  • f2022.c2022.p2022.t2022.nc
  • f2022.c2023.p2022.t2022.nc
    Ran 1/1 deferred expressions

-- Soil Thickness for Australian areas of intensive agriculture of Layer 1 (A Ho

-- Dataset ANZLIC ID ANZCW1202000149 --

Feature attribute definition Predicted average Thickness (mm) of soil layer 1
in the 0.01 X 0.01 degree quadrat.

Custodian: CSIRO Land & Water

Jurisdiction Australia

Short Description The digital map data is provided in geographical coordinates
based on the World Geodetic System 1984 (WGS84) datum. This raster data set has
a grid resolution of 0.001 degrees (approximately equivalent to 1.1 km).

The data set is a product of the National Land and Water Resources Audit
(NLWRA) as a base dataset.

Data Type: Spatial representation type RASTER

Projection Map: projection GEOGRAPHIC

Datum: WGS84

Map Units: DECIMAL DEGREES

Scale: Scale/ resolution 1:1 000 000

Usage Purpose Estimates of soil depths are needed to calculate the amount of
any soil constituent in either volume or mass terms (bulk density is also
needed) - for example, the volume of water stored in the rooting zone
potentially available for plant use, to assess total stores of soil carbon for
greenhouse inventory or to assess total stores of nutrients.

Provide indications of probable thickness soil layer 1 in agricultural areas
where soil thickness testing has not been carried out.

Use Limitation: This dataset is bound by the requirements set down by the
National Land & Water Resources Audit
To see the full metadata, call print_soil_thickness_metadata() in your R
session.

-- Soil Thickness for Australian areas of intensive agriculture of Layer 1 (A Ho

-- Dataset ANZLIC ID ANZCW1202000149 --

Feature attribute definition Predicted average Thickness (mm) of soil layer 1
in the 0.01 X 0.01 degree quadrat.

Custodian: CSIRO Land & Water

Jurisdiction Australia

Short Description The digital map data is provided in geographical coordinates
based on the World Geodetic System 1984 (WGS84) datum. This raster data set has
a grid resolution of 0.001 degrees (approximately equivalent to 1.1 km).

The data set is a product of the National Land and Water Resources Audit
(NLWRA) as a base dataset.

Data Type: Spatial representation type RASTER

Projection Map: projection GEOGRAPHIC

Datum: WGS84

Map Units: DECIMAL DEGREES

Scale: Scale/ resolution 1:1 000 000

Usage Purpose Estimates of soil depths are needed to calculate the amount of
any soil constituent in either volume or mass terms (bulk density is also
needed) - for example, the volume of water stored in the rooting zone
potentially available for plant use, to assess total stores of soil carbon for
greenhouse inventory or to assess total stores of nutrients.

Provide indications of probable thickness soil layer 1 in agricultural areas
where soil thickness testing has not been carried out.

Use Limitation: This dataset is bound by the requirements set down by the
National Land & Water Resources Audit
To see the full metadata, call print_soil_thickness_metadata() in your R
session.

-- Soil Thickness for Australian areas of intensive agriculture of Layer 1 (A Ho

-- Dataset ANZLIC ID ANZCW1202000149 --

Dataset ANZLIC ID ANZCW1202000149

Title Soil Thickness for Australian areas of intensive agriculture of Layer 1
(A Horizon - top-soil) (derived from soil mapping)

Custodian CSIRO, Land & Water

Jurisdiction Australia

Description Abstract Surface of predicted Thickness of soil layer 1 (A Horizon

  • top-soil) surface for the intensive agricultural areas of Australia. Data
    modelled from area based observations made by soil agencies both State and
    CSIRO and presented as .0.01 degree grid cells.

Topsoils (A horizons) are defined as the surface soil layers in which organic
matter accumulates, and may include dominantly organic surface layers (O and P
horizons).

The depth of topsoil is important because, with their higher organic matter
contents, topsoils (A horizon) generally have more suitable properties for
agriculture, including higher permeability and higher levels of soil nutrients.

Estimates of soil depths are needed to calculate the amount of any soil
constituent in either volume or mass terms (bulk density is also needed) - for
example, the volume of water stored in the rooting zone potentially available
for plant use, to assess total stores of soil carbon for Greenhouse inventory
or to assess total stores of nutrients.

The pattern of soil depth is strongly related to topography - the shape and
slope of the land. Deeper soils are typically found in the river valleys where
soils accumulate on floodplains and at the footslopes of ranges (zones of
deposition), while soils on hillslopes (zones of erosion) tend to be shallow.
Map of thickness of topsoil was derived from soil map data and interpreted
tables of soil properties for specific soil groups.

The quality of data on soil depth in existing soil profile datasets is
questionable and as the thickness of soil horizons varies locally with
topography, values for map units are general averages.

The final ASRIS polygon attributed surfaces are a mosaic of all of the data
obtained from various state and federal agencies. The surfaces have been
constructed with the best available soil survey information available at the
time. The surfaces also rely on a number of assumptions. One being that an area
weighted mean is a good estimate of the soil attributes for that polygon or
map-unit. Another assumption made is that the look-up tables provided by
McKenzie et al. (2000), state and territories accurately depict the soil
attribute values for each soil type.

The accuracy of the maps is most dependent on the scale of the original polygon
data sets and the level of soil survey that has taken place in each state. The
scale of the various soil maps used in deriving this map is available by
accessing the data-source grid, the scale is used as an assessment of the
likely accuracy of the modelling. The Atlas of Australian Soils is considered
to be the least accurate dataset and has therefore only been used where there
is no state based data. Of the state datasets Western Australian sub-systems,
South Australian land systems and NSW soil landscapes and reconnaissance
mapping would be the most reliable based on scale. NSW soil landscapes and
reconnaissance mapping use only one dominant soil type per polygon in the
estimation of attributes. South Australia and Western Australia use several
soil types per polygon or map-unit.

The digital map data is provided in geographical coordinates based on the World
Geodetic System 1984 (WGS84) datum. This raster data set has a grid resolution
of 0.001 degrees (approximately equivalent to 1.1 km).

The data set is a product of the National Land and Water Resources Audit
(NLWRA) as a base dataset.

Search Word(s) AGRICULTURE SOIL Physics Models

Geographic Extent Name(s) GEN Category

GEN Custodial Jurisdiction

GEN Name

Geographic Bounding Box North Bounding Latitude -10.707149 South Bounding
Latitude -43.516831 East Bounding Longitude 113.19673 West Bounding Longitude
153.990779

Geographic Extent Polygon(s) 115.0 -33.5,115.7 -33.3,115.7 -31.7,113.2
-26.2,113.5 -25.4,114.1 -26.4,114.3 -26.0,113.4 -24.3,114.1 -21.8,122.3
-18.2,122.2 -17.2,126.7 -13.6,129.1 -14.9,130.6 -12.3,132.6 -12.1,132.5
-11.6,131.9 -11.3,132.0 -11.1,137.0 -12.2,135.4 -14.7,140.0 -17.7,140.8
-17.4,141.7 -15.1,141.4 -13.7,142.2 -10.9,142.7 -10.7,143.9 -14.5,144.6
-14.1,145.3 -14.9,146.3 -18.8,148.9 -20.5,150.9 -22.6,153.2 -25.9,153.7
-28.8,153.0 -31.3,150.8 -34.8,150.0 -37.5,147.8 -37.9,146.3 -39.0,144.7
-38.4,143.5 -38.8,141.3 -38.4,139.7 -37.3,139.7 -36.9,139.9 -36.7,138.9
-35.5,138.1 -35.7,138.6 -34.7,138.1 -34.2,137.8 -35.1,136.9 -35.3,137.0
-34.9,137.5 -34.9,137.4 -34.0,137.9 -33.5,137.8 -32.6,137.3 -33.6,135.9
-34.7,136.1 -34.8,136.0 -35.0,135.1 -34.6,135.2 -34.5,135.4 -34.5,134.7
-33.3,134.0 -32.9,133.7 -32.1,133.3 -32.2,132.2 -32.0,131.3 -31.5,127.3
-32.3,126.0 -32.3,123.6 -33.9,123.2 -34.0,122.1 -34.0,121.9 -33.8,119.9
-34.0,119.6 -34.4,118.0 -35.1,116.0 -34.8,115.0 -34.3,115.0 -33.5

147.8 -42.9,147.9 -42.6,148.2 -42.1,148.3 -42.3,148.3 -41.3,148.3 -41.0,148.0
-40.7,147.4 -41.0,146.7 -41.1,146.6 -41.2,146.5 -41.1,146.4 -41.2,145.3
-40.8,145.3 -40.7,145.2 -40.8,145.2 -40.8,145.2 -40.8,145.0 -40.8,144.7
-40.7,144.7 -41.2,145.2 -42.2,145.4 -42.2,145.5 -42.4,145.5 -42.5,145.2
-42.3,145.5 -43.0,146.0 -43.3,146.0 -43.6,146.9 -43.6,146.9 -43.5,147.1
-43.3,147.0 -43.1,147.2 -43.3,147.3 -42.8,147.4 -42.9,147.6 -42.8,147.5
-42.8,147.8 -42.9,147.9 -43.0,147.7 -43.0,147.8 -43.2,147.9 -43.2,147.9
-43.2,148.0 -43.2,148.0 -43.1,148.0 -42.9,147.8 -42.9

136.7 -13.8,136.7 -13.7,136.6 -13.7,136.6 -13.8,136.4 -13.8,136.4 -14.1,136.3
-14.2,136.9 -14.3,137.0 -14.2,136.9 -14.2,136.7 -14.1,136.9 -13.8,136.7
-13.8,136.7 -13.8

139.5 -16.6,139.7 -16.5,139.4 -16.5,139.2 -16.7,139.3 -16.7,139.5 -16.6

153.0 -25.2,153.0 -25.7,153.1 -25.8,153.4 -25.0,153.2 -24.7,153.2 -25.0,153.0
-25.2

137.5 -36.1,137.7 -35.9,138.1 -35.9,137.9 -35.7,137.6 -35.7,137.6 -35.6,136.6
-35.8,136.7 -36.1,137.2 -36.0,137.5 -36.1

143.9 -39.7,144.0 -39.6,144.1 -39.8,143.9 -40.2,143.9 -40.0,143.9 -39.7

148.0 -39.7,147.7 -39.9,147.9 -39.9,148.0 -40.1,148.1 -40.3,148.3 -40.2,148.3
-40.0,148.0 -39.7

148.1 -40.4,148.0 -40.4,148.4 -40.3,148.4 -40.5,148.1 -40.4

130.4 -11.3,130.4 -11.2,130.6 -11.3,130.7 -11.4,130.9 -11.3,131.0 -11.4,131.1
-11.3,131.2 -11.4,131.3 -11.2,131.5 -11.4,131.5 -11.5,131.0 -11.9,130.8
-11.8,130.6 -11.7,130.0 -11.8,130.1 -11.7,130.3 -11.7,130.1 -11.5,130.4 -11.3

Data Currency Beginning date 1999-09-01

Ending date 2001-03-31

Dataset Status Progress COMPLETE

Maintenance and Update Frequency NOT PLANNED

Access Stored Data Format DIGITAL - ESRI Arc/Info integer GRID

Available Format Type DIGITAL - ESRI Arc/Info integer GRID

Access Constraint Subject to the terms & condition of the data access &
management agreement between the National Land & Water Audit and ANZLIC parties

Data Quality Lineage The soil attribute surface was created using the following
datasets 1. The digital polygon coverage of the Soil-Landforms of the Murray
Darling Basis (MDBSIS)(Bui et al. 1998), classified as principal profile forms
(PPF's) (Northcote 1979). 2. The digital Atlas of Australian Soils (Northcote
et al.1960-1968)(Leahy, 1993). 3. Western Australia land systems coverage
(Agriculture WA). 4. Western Australia sub-systems coverage (Agriculture WA).
5. Ord river catchment soils coverage (Agriculture WA). 6. Victoria soils
coverage (Victorian Department of Natural Resources and Environment - NRE). 7.
NSW Soil Landscapes and reconnaissance soil landscape mapping (NSW Department
of Land and Water Conservation - DLWC). 8. New South Wales Land systems west
(NSW Department of Land and Water Conservation - DLWC). 9. South Australia soil
land-systems (Primary Industries and Resources South Australia - PIRSA). 10.
Northern Territory soils coverage (Northern Territory Department of Lands,
Planning and Environment). 11. A mosaic of Queensland soils coverages
(Queensland Department of Natural Resources - QDNR). 12. A look-up table
linking PPF values from the Atlas of Australian Soils with interpreted soil
attributes (McKenzie et al. 2000). 13. Look_up tables provided by WA
Agriculture linking WA soil groups with interpreted soil attributes. 14.
Look_up tables provided by PIRSA linking SA soil groups with interpreted soil
attributes.

The continuous raster surface representing Thickness of soil layer 1 was
created by combining national and state level digitised land systems maps and
soil surveys linked to look-up tables listing soil type and corresponding
attribute values.

Because thickness is used sparingly in the Factual Key, estimations of
thickness in the look-up tables were made using empirical correlations for
particular soil types.

To estimate a soil attribute where more than one soil type was given for a
polygon or map-unit, the soil attribute values related to each soil type in the
look-up table were weighted according to the area occupied by that soil type
within the polygon or map-unit. The final soil attribute values are an area
weighted average for a polygon or map-unit. The polygon data was then
converted to a continuous raster surface using the soil attribute values
calculated for each polygon.

The ASRIS soil attribute surfaces created using polygon attribution relied on a
number of data sets from various state agencies. Each polygon data set was
turned into a continuous surface grid based on the calculated soil attribute
value for that polygon. The grids where then merged on the basis that, where
available, state data replaced the Atlas of Australian Soils and MDBSIS.
MDBSIS derived soil attribute values were restricted to areas where MDBSIS was
deemed to be more accurate that the Atlas of Australian Soils (see Carlile et
al (2001a).

In cases where a soil type was missing from the look-up table or layer 2 did
not exist for that soil type, the percent area of the soils remaining were
adjusted prior to calculating the final soil attribute value. The method used
to attribute polygons was dependent on the data supplied by individual State
agencies.

The modelled grid was resampled from 0.0025 degree cells to 0.01 degree cells
using bilinear interpolation

Positional Accuracy The predictive surface is a 0.01 X 0.01 degree grid and has
a locational accurate of about 1m.

The positional accuracy of the defining polygons have variable positional
accuracy most locations are expected to be within 100m of the recorded
location. The vertical accuracy is not relevant. The positional assessment has
been made by considering the tools used to generate the locational information
and contacting the data providers.

The other parameters used in the production of the led surface have a range of
positional accuracy ranging from + - 50 m to + - kilometres. This contribute
to the loss of attribute accuracy in the surface.

Attribute Accuracy Input attribute accuracy for the areas is highly variable.
The predictive has a variable and much lower attribute accuracy due to the
irregular distribution and the limited positional accuracy of the parameters
used for modelling.

There are several sources of error in estimating soil depth and thickness of
horizons for the look-up tables. Because thickness is used sparingly in the
Factual Key, estimations of thickness in the look-up tables were made using
empirical correlations for particular soil types. The quality of data on soil
depth in existing soil profile datasets is questionable, in soil mapping,
thickness of soil horizons varies locally with topography, so values for map
units are general averages. The definition of the depth of soil or regolith is
imprecise and it can be difficult to determine the lower limit of soil.

The assumption made that an area weighted mean of soil attribute values based
on soil type is a good estimation of a soil property is debatable, in that it
does not supply the soil attribute value at any given location. Rather it is
designed to show national and regional patterns in soil properties. The use of
the surfaces at farm or catchment scale modelling may prove inaccurate. Also
the use of look-up tables to attribute soil types is only as accurate as the
number of observations used to estimate a attribute value for a soil type. Some
soil types in the look-up tables may have few observations, yet the average
attribute value is still taken as the attribute value for that soil type.
Different states are using different taxonomic schemes making a national soil
database difficult. Another downfall of the area weighted approach is that some
soil types may not be listed in look-up tables. If a soil type is a dominant
one within a polygon or map-unit, but is not listed within the look-up table or
is not attributed within the look-up table then the final soil attribute value
for that polygon will be biased towards the minor soil types that do exist.
This may also happen when a large area is occupied by a soil type which has no
B horizon. In this case the final soil attribute value will be area weighted on
the soils with a B horizon, ignoring a major soil type within that polygon or
map-unit. The layer 2 surfaces have large areas of no-data because all soils
listed for a particular map-unit or polygon had no B horizon.

Logical Consistency Surface is fully logically consistent as only one parameter
is shown, as predicted average Soil Thickness within each grid cell

Completeness Surface is nearly complete. There are some areas (about %1
missing) for which insufficient parameters were known to provide a useful
prediction and thus attributes are absent in these areas.

Contact Information Contact Organisation (s) CSIRO, Land & Water

Contact Position Project Leader

Mail Address ACLEP, GPO 1666

Locality Canberra

State ACT

Country AUSTRALIA

Postcode 2601

Telephone 02 6246 5922

Facsimile 02 6246 5965

Electronic Mail Address [email protected]

Metadata Date Metadata Date 2001-07-01

Additional Metadata Additional Metadata

Entity and Attributes Entity Name Soil Thickness Layer 1 (derived from mapping)

Entity description Estimated Soil Thickness (mm) of Layer 1 on a cell by cell
basis

Feature attribute name VALUE

Feature attribute definition Predicted average Thickness (mm) of soil layer 1
in the 0.01 X 0.01 degree quadrat

Data Type Spatial representation type RASTER

Projection Map projection GEOGRAPHIC

Datum WGS84

Map units DECIMAL DEGREES

Scale Scale/ resolution 1:1 000 000

Usage Purpose Estimates of soil depths are needed to calculate the amount of
any soil constituent in either volume or mass terms (bulk density is also
needed) - for example, the volume of water stored in the rooting zone
potentially available for plant use, to assess total stores of soil carbon for
Greenhouse inventory or to assess total stores of nutrients.

Provide indications of probable Thickness soil layer 1 in agricultural areas
where soil thickness testing has not been carried out

Use Use Limitation This dataset is bound by the requirements set down by the
National Land & Water Resources Audit

-- Soil Thickness for Australian areas of intensive agriculture of Layer 1 (A Ho

-- Dataset ANZLIC ID ANZCW1202000149 --

Dataset ANZLIC ID ANZCW1202000149

Title Soil Thickness for Australian areas of intensive agriculture of Layer 1
(A Horizon - top-soil) (derived from soil mapping)

Custodian CSIRO, Land & Water

Jurisdiction Australia

Description Abstract Surface of predicted Thickness of soil layer 1 (A Horizon

  • top-soil) surface for the intensive agricultural areas of Australia. Data
    modelled from area based observations made by soil agencies both State and
    CSIRO and presented as .0.01 degree grid cells.

Topsoils (A horizons) are defined as the surface soil layers in which organic
matter accumulates, and may include dominantly organic surface layers (O and P
horizons).

The depth of topsoil is important because, with their higher organic matter
contents, topsoils (A horizon) generally have more suitable properties for
agriculture, including higher permeability and higher levels of soil nutrients.

Estimates of soil depths are needed to calculate the amount of any soil
constituent in either volume or mass terms (bulk density is also needed) - for
example, the volume of water stored in the rooting zone potentially available
for plant use, to assess total stores of soil carbon for Greenhouse inventory
or to assess total stores of nutrients.

The pattern of soil depth is strongly related to topography - the shape and
slope of the land. Deeper soils are typically found in the river valleys where
soils accumulate on floodplains and at the footslopes of ranges (zones of
deposition), while soils on hillslopes (zones of erosion) tend to be shallow.
Map of thickness of topsoil was derived from soil map data and interpreted
tables of soil properties for specific soil groups.

The quality of data on soil depth in existing soil profile datasets is
questionable and as the thickness of soil horizons varies locally with
topography, values for map units are general averages.

The final ASRIS polygon attributed surfaces are a mosaic of all of the data
obtained from various state and federal agencies. The surfaces have been
constructed with the best available soil survey information available at the
time. The surfaces also rely on a number of assumptions. One being that an area
weighted mean is a good estimate of the soil attributes for that polygon or
map-unit. Another assumption made is that the look-up tables provided by
McKenzie et al. (2000), state and territories accurately depict the soil
attribute values for each soil type.

The accuracy of the maps is most dependent on the scale of the original polygon
data sets and the level of soil survey that has taken place in each state. The
scale of the various soil maps used in deriving this map is available by
accessing the data-source grid, the scale is used as an assessment of the
likely accuracy of the modelling. The Atlas of Australian Soils is considered
to be the least accurate dataset and has therefore only been used where there
is no state based data. Of the state datasets Western Australian sub-systems,
South Australian land systems and NSW soil landscapes and reconnaissance
mapping would be the most reliable based on scale. NSW soil landscapes and
reconnaissance mapping use only one dominant soil type per polygon in the
estimation of attributes. South Australia and Western Australia use several
soil types per polygon or map-unit.

The digital map data is provided in geographical coordinates based on the World
Geodetic System 1984 (WGS84) datum. This raster data set has a grid resolution
of 0.001 degrees (approximately equivalent to 1.1 km).

The data set is a product of the National Land and Water Resources Audit
(NLWRA) as a base dataset.

Search Word(s) AGRICULTURE SOIL Physics Models

Geographic Extent Name(s) GEN Category

GEN Custodial Jurisdiction

GEN Name

Geographic Bounding Box North Bounding Latitude -10.707149 South Bounding
Latitude -43.516831 East Bounding Longitude 113.19673 West Bounding Longitude
153.990779

Geographic Extent Polygon(s) 115.0 -33.5,115.7 -33.3,115.7 -31.7,113.2
-26.2,113.5 -25.4,114.1 -26.4,114.3 -26.0,113.4 -24.3,114.1 -21.8,122.3
-18.2,122.2 -17.2,126.7 -13.6,129.1 -14.9,130.6 -12.3,132.6 -12.1,132.5
-11.6,131.9 -11.3,132.0 -11.1,137.0 -12.2,135.4 -14.7,140.0 -17.7,140.8
-17.4,141.7 -15.1,141.4 -13.7,142.2 -10.9,142.7 -10.7,143.9 -14.5,144.6
-14.1,145.3 -14.9,146.3 -18.8,148.9 -20.5,150.9 -22.6,153.2 -25.9,153.7
-28.8,153.0 -31.3,150.8 -34.8,150.0 -37.5,147.8 -37.9,146.3 -39.0,144.7
-38.4,143.5 -38.8,141.3 -38.4,139.7 -37.3,139.7 -36.9,139.9 -36.7,138.9
-35.5,138.1 -35.7,138.6 -34.7,138.1 -34.2,137.8 -35.1,136.9 -35.3,137.0
-34.9,137.5 -34.9,137.4 -34.0,137.9 -33.5,137.8 -32.6,137.3 -33.6,135.9
-34.7,136.1 -34.8,136.0 -35.0,135.1 -34.6,135.2 -34.5,135.4 -34.5,134.7
-33.3,134.0 -32.9,133.7 -32.1,133.3 -32.2,132.2 -32.0,131.3 -31.5,127.3
-32.3,126.0 -32.3,123.6 -33.9,123.2 -34.0,122.1 -34.0,121.9 -33.8,119.9
-34.0,119.6 -34.4,118.0 -35.1,116.0 -34.8,115.0 -34.3,115.0 -33.5

147.8 -42.9,147.9 -42.6,148.2 -42.1,148.3 -42.3,148.3 -41.3,148.3 -41.0,148.0
-40.7,147.4 -41.0,146.7 -41.1,146.6 -41.2,146.5 -41.1,146.4 -41.2,145.3
-40.8,145.3 -40.7,145.2 -40.8,145.2 -40.8,145.2 -40.8,145.0 -40.8,144.7
-40.7,144.7 -41.2,145.2 -42.2,145.4 -42.2,145.5 -42.4,145.5 -42.5,145.2
-42.3,145.5 -43.0,146.0 -43.3,146.0 -43.6,146.9 -43.6,146.9 -43.5,147.1
-43.3,147.0 -43.1,147.2 -43.3,147.3 -42.8,147.4 -42.9,147.6 -42.8,147.5
-42.8,147.8 -42.9,147.9 -43.0,147.7 -43.0,147.8 -43.2,147.9 -43.2,147.9
-43.2,148.0 -43.2,148.0 -43.1,148.0 -42.9,147.8 -42.9

136.7 -13.8,136.7 -13.7,136.6 -13.7,136.6 -13.8,136.4 -13.8,136.4 -14.1,136.3
-14.2,136.9 -14.3,137.0 -14.2,136.9 -14.2,136.7 -14.1,136.9 -13.8,136.7
-13.8,136.7 -13.8

139.5 -16.6,139.7 -16.5,139.4 -16.5,139.2 -16.7,139.3 -16.7,139.5 -16.6

153.0 -25.2,153.0 -25.7,153.1 -25.8,153.4 -25.0,153.2 -24.7,153.2 -25.0,153.0
-25.2

137.5 -36.1,137.7 -35.9,138.1 -35.9,137.9 -35.7,137.6 -35.7,137.6 -35.6,136.6
-35.8,136.7 -36.1,137.2 -36.0,137.5 -36.1

143.9 -39.7,144.0 -39.6,144.1 -39.8,143.9 -40.2,143.9 -40.0,143.9 -39.7

148.0 -39.7,147.7 -39.9,147.9 -39.9,148.0 -40.1,148.1 -40.3,148.3 -40.2,148.3
-40.0,148.0 -39.7

148.1 -40.4,148.0 -40.4,148.4 -40.3,148.4 -40.5,148.1 -40.4

130.4 -11.3,130.4 -11.2,130.6 -11.3,130.7 -11.4,130.9 -11.3,131.0 -11.4,131.1
-11.3,131.2 -11.4,131.3 -11.2,131.5 -11.4,131.5 -11.5,131.0 -11.9,130.8
-11.8,130.6 -11.7,130.0 -11.8,130.1 -11.7,130.3 -11.7,130.1 -11.5,130.4 -11.3

Data Currency Beginning date 1999-09-01

Ending date 2001-03-31

Dataset Status Progress COMPLETE

Maintenance and Update Frequency NOT PLANNED

Access Stored Data Format DIGITAL - ESRI Arc/Info integer GRID

Available Format Type DIGITAL - ESRI Arc/Info integer GRID

Access Constraint Subject to the terms & condition of the data access &
management agreement between the National Land & Water Audit and ANZLIC parties

Data Quality Lineage The soil attribute surface was created using the following
datasets 1. The digital polygon coverage of the Soil-Landforms of the Murray
Darling Basis (MDBSIS)(Bui et al. 1998), classified as principal profile forms
(PPF's) (Northcote 1979). 2. The digital Atlas of Australian Soils (Northcote
et al.1960-1968)(Leahy, 1993). 3. Western Australia land systems coverage
(Agriculture WA). 4. Western Australia sub-systems coverage (Agriculture WA).
5. Ord river catchment soils coverage (Agriculture WA). 6. Victoria soils
coverage (Victorian Department of Natural Resources and Environment - NRE). 7.
NSW Soil Landscapes and reconnaissance soil landscape mapping (NSW Department
of Land and Water Conservation - DLWC). 8. New South Wales Land systems west
(NSW Department of Land and Water Conservation - DLWC). 9. South Australia soil
land-systems (Primary Industries and Resources South Australia - PIRSA). 10.
Northern Territory soils coverage (Northern Territory Department of Lands,
Planning and Environment). 11. A mosaic of Queensland soils coverages
(Queensland Department of Natural Resources - QDNR). 12. A look-up table
linking PPF values from the Atlas of Australian Soils with interpreted soil
attributes (McKenzie et al. 2000). 13. Look_up tables provided by WA
Agriculture linking WA soil groups with interpreted soil attributes. 14.
Look_up tables provided by PIRSA linking SA soil groups with interpreted soil
attributes.

The continuous raster surface representing Thickness of soil layer 1 was
created by combining national and state level digitised land systems maps and
soil surveys linked to look-up tables listing soil type and corresponding
attribute values.

Because thickness is used sparingly in the Factual Key, estimations of
thickness in the look-up tables were made using empirical correlations for
particular soil types.

To estimate a soil attribute where more than one soil type was given for a
polygon or map-unit, the soil attribute values related to each soil type in the
look-up table were weighted according to the area occupied by that soil type
within the polygon or map-unit. The final soil attribute values are an area
weighted average for a polygon or map-unit. The polygon data was then
converted to a continuous raster surface using the soil attribute values
calculated for each polygon.

The ASRIS soil attribute surfaces created using polygon attribution relied on a
number of data sets from various state agencies. Each polygon data set was
turned into a continuous surface grid based on the calculated soil attribute
value for that polygon. The grids where then merged on the basis that, where
available, state data replaced the Atlas of Australian Soils and MDBSIS.
MDBSIS derived soil attribute values were restricted to areas where MDBSIS was
deemed to be more accurate that the Atlas of Australian Soils (see Carlile et
al (2001a).

In cases where a soil type was missing from the look-up table or layer 2 did
not exist for that soil type, the percent area of the soils remaining were
adjusted prior to calculating the final soil attribute value. The method used
to attribute polygons was dependent on the data supplied by individual State
agencies.

The modelled grid was resampled from 0.0025 degree cells to 0.01 degree cells
using bilinear interpolation

Positional Accuracy The predictive surface is a 0.01 X 0.01 degree grid and has
a locational accurate of about 1m.

The positional accuracy of the defining polygons have variable positional
accuracy most locations are expected to be within 100m of the recorded
location. The vertical accuracy is not relevant. The positional assessment has
been made by considering the tools used to generate the locational information
and contacting the data providers.

The other parameters used in the production of the led surface have a range of
positional accuracy ranging from + - 50 m to + - kilometres. This contribute
to the loss of attribute accuracy in the surface.

Attribute Accuracy Input attribute accuracy for the areas is highly variable.
The predictive has a variable and much lower attribute accuracy due to the
irregular distribution and the limited positional accuracy of the parameters
used for modelling.

There are several sources of error in estimating soil depth and thickness of
horizons for the look-up tables. Because thickness is used sparingly in the
Factual Key, estimations of thickness in the look-up tables were made using
empirical correlations for particular soil types. The quality of data on soil
depth in existing soil profile datasets is questionable, in soil mapping,
thickness of soil horizons varies locally with topography, so values for map
units are general averages. The definition of the depth of soil or regolith is
imprecise and it can be difficult to determine the lower limit of soil.

The assumption made that an area weighted mean of soil attribute values based
on soil type is a good estimation of a soil property is debatable, in that it
does not supply the soil attribute value at any given location. Rather it is
designed to show national and regional patterns in soil properties. The use of
the surfaces at farm or catchment scale modelling may prove inaccurate. Also
the use of look-up tables to attribute soil types is only as accurate as the
number of observations used to estimate a attribute value for a soil type. Some
soil types in the look-up tables may have few observations, yet the average
attribute value is still taken as the attribute value for that soil type.
Different states are using different taxonomic schemes making a national soil
database difficult. Another downfall of the area weighted approach is that some
soil types may not be listed in look-up tables. If a soil type is a dominant
one within a polygon or map-unit, but is not listed within the look-up table or
is not attributed within the look-up table then the final soil attribute value
for that polygon will be biased towards the minor soil types that do exist.
This may also happen when a large area is occupied by a soil type which has no
B horizon. In this case the final soil attribute value will be area weighted on
the soils with a B horizon, ignoring a major soil type within that polygon or
map-unit. The layer 2 surfaces have large areas of no-data because all soils
listed for a particular map-unit or polygon had no B horizon.

Logical Consistency Surface is fully logically consistent as only one parameter
is shown, as predicted average Soil Thickness within each grid cell

Completeness Surface is nearly complete. There are some areas (about %1
missing) for which insufficient parameters were known to provide a useful
prediction and thus attributes are absent in these areas.

Contact Information Contact Organisation (s) CSIRO, Land & Water

Contact Position Project Leader

Mail Address ACLEP, GPO 1666

Locality Canberra

State ACT

Country AUSTRALIA

Postcode 2601

Telephone 02 6246 5922

Facsimile 02 6246 5965

Electronic Mail Address [email protected]

Metadata Date Metadata Date 2001-07-01

Additional Metadata Additional Metadata

Entity and Attributes Entity Name Soil Thickness Layer 1 (derived from mapping)

Entity description Estimated Soil Thickness (mm) of Layer 1 on a cell by cell
basis

Feature attribute name VALUE

Feature attribute definition Predicted average Thickness (mm) of soil layer 1
in the 0.01 X 0.01 degree quadrat

Data Type Spatial representation type RASTER

Projection Map projection GEOGRAPHIC

Datum WGS84

Map units DECIMAL DEGREES

Scale Scale/ resolution 1:1 000 000

Usage Purpose Estimates of soil depths are needed to calculate the amount of
any soil constituent in either volume or mass terms (bulk density is also
needed) - for example, the volume of water stored in the rooting zone
potentially available for plant use, to assess total stores of soil carbon for
Greenhouse inventory or to assess total stores of nutrients.

Provide indications of probable Thickness soil layer 1 in agricultural areas
where soil thickness testing has not been carried out

Use Use Limitation This dataset is bound by the requirements set down by the
National Land & Water Resources Audit
Ran 1/1 deferred expressions
There do not appear to be any files cached for {read.abares}.
There do not appear to be any files cached for {read.abares}.

-- Locally Available {read.abares} Cached Files --------------------------------

  • test.R

-- Locally Available {read.abares} Cached Files --------------------------------

  • test.R
    Ran 1/1 deferred expressions
    Will return stars object with 612226 cells.
    No projection information found in nc file.
    Coordinate variable units found to be degrees,
    assuming WGS84 Lat/Lon.
    Killed

R CMD check generated the following check_fail:

  1. rcmdcheck_tests_pass

Test coverage with covr

ERROR: Test Coverage Failed

Cyclocomplexity with cyclocomp

No functions have cyclocomplexity >= 15

Static code analyses with lintr

lintr found no issues with this package!


4. Other Checks

Details of other checks (click to open)

✖️ The following function name is duplicated in other packages:

    • clear_cache from catalog, lintr, quincunx, rmonad


Package Versions

package version
pkgstats 0.2.0.46
pkgcheck 0.1.2.63


Editor-in-Chief Instructions:

Processing may not proceed until the items marked with ✖️ have been resolved.

@adamhsparks
Copy link
Member Author

@ropensci-review-bot check package

@ropensci-review-bot
Copy link
Collaborator

Thanks, about to send the query.

@ropensci-review-bot
Copy link
Collaborator

🚀

Editor check started

👋

@mpadge
Copy link
Member

mpadge commented Nov 6, 2024

@adamhsparks The tests on our machines just keep running for hours and hours and days even. Can you check on your side to make sure there's not some flag that works on GitHub machines to prevent this, yet might not stop tests on our machines?

@adamhsparks
Copy link
Member Author

adamhsparks commented Nov 6, 2024

I’ve got nothing. It only takes a few minutes on GH now. 🤷

Screenshot 2024-11-07 at 06 44 47

@adamhsparks
Copy link
Member Author

I'll just close this for now at least. Unsure how to fix issues with package checks and don't have the mental bandwidth to deal with it right now.

@adamhsparks
Copy link
Member Author

@ropensci-review-bot check package

@adamhsparks adamhsparks reopened this Nov 15, 2024
@ropensci-review-bot
Copy link
Collaborator

Thanks, about to send the query.

@ropensci-review-bot
Copy link
Collaborator

🚀

Editor check started

👋

@adamhsparks
Copy link
Member Author

@ropensci-review-bot check package

@ropensci-review-bot
Copy link
Collaborator

Thanks, about to send the query.

@ropensci-review-bot
Copy link
Collaborator

🚀

Editor check started

👋

@mpadge
Copy link
Member

mpadge commented Nov 19, 2024

Sorry @adamhsparks, I'll try to find out the problem with the checks asap and get back to you ... 😕

@ropensci-review-bot
Copy link
Collaborator

Checks for read.abares (v0.1.0)

git hash: a7a41aab

  • ✔️ Package name is available
  • ✔️ has a 'codemeta.json' file.
  • ✔️ has a 'contributing' file.
  • ✔️ uses 'roxygen2'.
  • ✔️ 'DESCRIPTION' has a URL field.
  • ✔️ 'DESCRIPTION' has a BugReports field.
  • ✔️ Package has at least one HTML vignette
  • ✔️ All functions have examples.
  • ✔️ Package has continuous integration checks.
  • ✖️ Package coverage is 5.3% (should be at least 75%).
  • ✔️ R CMD check found no errors.
  • ✔️ R CMD check found no warnings.
  • 👀 Function names are duplicated in other packages

Important: All failing checks above must be addressed prior to proceeding

(Checks marked with 👀 may be optionally addressed.)

Package License: MIT + file LICENSE


1. Package Dependencies

Details of Package Dependency Usage (click to open)

The table below tallies all function calls to all packages ('ncalls'), both internal (r-base + recommended, along with the package itself), and external (imported and suggested packages). 'NA' values indicate packages to which no identified calls to R functions could be found. Note that these results are generated by an automated code-tagging system which may not be entirely accurate.

type package ncalls
internal base 76
internal read.abares 9
internal methods 3
internal stats 3
internal tools 2
internal utils 2
imports data.table 21
imports purrr 6
imports httr2 5
imports terra 4
imports openxlsx2 3
imports sf 3
imports stars 3
imports tidync 3
imports stringr 2
imports readtext 1
imports cli NA
imports lubridate NA
imports withr NA
suggests knitr NA
suggests pander NA
suggests rmarkdown NA
suggests roxyglobals NA
suggests spelling NA
suggests testthat NA
linking_to NA NA

Click below for tallies of functions used in each package. Locations of each call within this package may be generated locally by running 's <- pkgstats::pkgstats(<path/to/repo>)', and examining the 'external_calls' table.

base

file.path (32), tempdir (13), c (12), dirname (7), url (4), lapply (2), list (2), list.files (2), append (1), nchar (1)

data.table

fread (9), fifelse (7), as.data.table (3), rbindlist (1), setcolorder (1)

read.abares

clear_cache (1), get_aagis_regions (1), get_abares_trade (1), get_abares_trade_regions (1), get_agfd (1), get_historical_forecast_database (1), get_historical_national_estimates (1), get_soil_thickness (1), inspect_cache (1)

purrr

map (4), modify_depth (1), quietly (1)

httr2

req_options (2), request (2), req_retry (1)

terra

rast (4)

methods

new (3)

openxlsx2

read_xlsx (3)

sf

read_sf (2), st_read (1)

stars

read_ncdf (2), read_stars (1)

stats

var (2), dt (1)

tidync

tidync (2), hyper_tibble (1)

stringr

str_locate (1), str_sub (1)

tools

R_user_dir (2)

utils

unzip (2)

readtext

readtext (1)

NOTE: Some imported packages appear to have no associated function calls; please ensure with author that these 'Imports' are listed appropriately.


2. Statistical Properties

This package features some noteworthy statistical properties which may need to be clarified by a handling editor prior to progressing.

Details of statistical properties (click to open)

The package has:

  • code in R (100% in 23 files) and
  • 1 authors
  • 1 vignette
  • no internal data file
  • 13 imported packages
  • 26 exported functions (median 9 lines of code)
  • 49 non-exported functions in R (median 16 lines of code)

Statistical properties of package structure as distributional percentiles in relation to all current CRAN packages
The following terminology is used:

  • loc = "Lines of Code"
  • fn = "function"
  • exp/not_exp = exported / not exported

All parameters are explained as tooltips in the locally-rendered HTML version of this report generated by the checks_to_markdown() function

The final measure (fn_call_network_size) is the total number of calls between functions (in R), or more abstract relationships between code objects in other languages. Values are flagged as "noteworthy" when they lie in the upper or lower 5th percentile.

measure value percentile noteworthy
files_R 23 83.4
files_vignettes 2 81.7
files_tests 22 95.6
loc_R 628 52.5
loc_vignettes 265 56.8
loc_tests 893 81.8
num_vignettes 1 58.9
n_fns_r 75 67.2
n_fns_r_exported 26 73.8
n_fns_r_not_exported 49 65.1
n_fns_per_file_r 2 34.9
num_params_per_fn 1 1.8 TRUE
loc_per_fn_r 11 32.9
loc_per_fn_r_exp 9 19.8
loc_per_fn_r_not_exp 16 53.0
rel_whitespace_R 20 56.4
rel_whitespace_vignettes 50 70.4
rel_whitespace_tests 10 66.2
doclines_per_fn_exp 25 23.7
doclines_per_fn_not_exp 0 0.0 TRUE
fn_call_network_size 36 57.9

2a. Network visualisation

Click to see the interactive network visualisation of calls between objects in package


3. goodpractice and other checks

Details of goodpractice checks (click to open)

3a. Continuous Integration Badges

R-CMD-check.yaml

GitHub Workflow Results

id name conclusion sha run_number date
11905521413 pages build and deployment success 55bec0 29 2024-11-19
11905498648 pkgdown.yaml success a7a41a 31 2024-11-19
11905498644 R-CMD-check.yaml success a7a41a 31 2024-11-19
11693994081 test-coverage.yaml failure bc173b 27 2024-11-05

3b. goodpractice results

R CMD check with rcmdcheck

rcmdcheck found no errors, warnings, or notes

Test coverage with covr

Package coverage: 5.35

The following files are not completely covered by tests:

file coverage
R/get_aagis_regions.R 0%
R/get_abares_trade_regions.R 0%
R/get_abares_trade.R 0%
R/get_agfd.R 0%
R/get_estimates_by_performance_category.R 0%
R/get_estimates_by_size.R 0%
R/get_historical_forecast_database.R 0%
R/get_historical_national_estimates.R 0%
R/get_historical_regional_estimates.R 0%
R/get_historical_state_estimates.R 0%
R/get_soil_thickness.R 0%
R/internal_functions.R 10.71%
R/read_agfd_dt.R 0%
R/read_agfd_stars.R 0%
R/read_agfd_terra.R 0%
R/read_agfd_tidync.R 0%
R/read_soil_thickness_stars.R 0%
R/read_soil_thickness_terra.R 0%

Cyclocomplexity with cyclocomp

No functions have cyclocomplexity >= 15

Static code analyses with lintr

lintr found no issues with this package!


4. Other Checks

Details of other checks (click to open)

✖️ The following function name is duplicated in other packages:

    • clear_cache from catalog, lintr, quincunx, rmonad


Package Versions

package version
pkgstats 0.2.0.48
pkgcheck 0.1.2.68


Editor-in-Chief Instructions:

Processing may not proceed until the items marked with ✖️ have been resolved.

@adamhsparks
Copy link
Member Author

adamhsparks commented Nov 20, 2024

@emilyriederer, the test coverage is >90%, see attached screenshot of local covr::report().

However, there are issues with connectivity between GitHub CI and ABARES where GitHub just won't download some of the data. From what @mpadge and I have sussed out, it's a local issue (GitHub) and I can't really troubleshoot their setup. I'd note here that this also seems to happen on some of the smaller files that are just CSVs for some reason, so it doesn't appear to be confined to large files only. So I've opted to skip most of the tests on CI using skip_on_ci() for any function that downloads data. But I'm open to suggestions as to how to improve this. It's just one reason I've submitted it for review. 😃

Screenshot 2024-11-20 at 09 43 52

@emilyriederer
Copy link

Thanks @adamhsparks !

It looks like {covr} also has options to explicitly exclude certain functions from code coverage calculations (details here). Would it make sense to use that logic just so our bot and the local review can have consistent results? (curious @mpadge thoughts also)

I'm fairly comfortable moving forward either way; just one thought to flesh out best practices here.

@adamhsparks
Copy link
Member Author

Thanks, @emilyriederer, I'd prefer not to exclude these functions as using testing helped me squash a few bugs last week and there's no reason to exclude them in the package where I might normally consider excluding functions. This appears to be a technical limitation outside of the package that would hinder the package's quality if implemented.

@mpadge
Copy link
Member

mpadge commented Nov 25, 2024

@emilyriederer I'm happy to accept Adam's explanation there. We do require coverage reports, but have no policies against locally-generated reports like used here. I think this would represent the first submission to do so, and so would maybe only ask @adamhsparks if you could have a look over the relevant section of the dev guide to see whether anything might be clarified or improved there?

@adamhsparks
Copy link
Member Author

Thanks, @mpadge. I'll have a look at that.

I was reflecting while walking home from work. I can probably include small samples of the data in the package for testing purposes and examples as they are CC By licensed. That might get me to 75% coverage with CI.

I'll investigate further.

@mpadge
Copy link
Member

mpadge commented Nov 25, 2024

For non-binary files (your csvs and maybe more?), you can also use httptest2 redactor functions to find some early pattern in a response and gsub the rest away. I use a lot of those like here to record only tiny portions of otherwise potentially huge response files.

@emilyriederer
Copy link

Thanks, @adamhsparks ! That sounds like a great suggestion. I'll hold if you want to investigate including sample data or using @mpadge 's idea. If those prove intractable, I'm also comfortable with moving checks offline / local, so please feel free to ping and we can move the ball forward here.

@adamhsparks
Copy link
Member Author

Yeah, let’s hold. I may be a while though with holidays approaching.

@maurolepore
Copy link
Member

@ropensci-review-bot assign @obrl-soil as editor

@ropensci-review-bot
Copy link
Collaborator

Assigned! @obrl-soil is now the editor

@maurolepore
Copy link
Member

@obrl-soil thanks so much for accepting to handle this submission. Feel free to reach out to me here, slack or email with any question you may have.

@obrl-soil
Copy link

Hi @adamhsparks - I'm unfortunately having a download-timeout problem similar to the discussion about CI issues above, only on my local machine (Surface laptop running win 11). I can download the target files via the browser just fine, and I can't see anything of note when I run httr2::req_perform(., verbosity = 3). Not sure where to go from here. Maybe the httr2 maintainers might have some tips?

@maurolepore
Copy link
Member

@obrl-soil, just to show I'm watching so you feel our support. Adam or me will come back soon too help.

@adamhsparks
Copy link
Member Author

Hi @obrl-soil, which file is causing the issue?

Unfortunately, I'm unable to duplicate, it's passing all the tests for me locally on my Mac, which means the files are available and downloading through R. With the exception of GitHub CI which isn't really much help in troubleshooting any issues, itworksforme. 🙈

I can see if I can get one of the team to try some of the functions tomorrow or running tests on Windows laptops at work, maybe?

@adamhsparks
Copy link
Member Author

@obrl-soil, can you provide me with any more details? I've gone through some of the functions with one of the team here at Curtin on a fresh R install on Windows and it downloads files without issues, so best I can tell it's not an OS-level problem.

@obrl-soil
Copy link

I'd be surprised if it was, this feels like more of a curl/httr/browser(??) type problem. My default browser is Firefox, but I also tried this on a much more locked down work machine (win10 and Edge only) with the same result.

I tried blowing away my entire R/RTools/RStudio setup on my home machine tonight and started from scratch. No luck, but here's the check logs:

> library(devtools)
Loading required package: usethis
> check()
══ Documenting ════════════════════════════════════════════════════════════════════════════
ℹ Updating read.abares documentation
ℹ Loading read.abares
✖ get_soil_thickness.R:175: @description refers to unavailable topic pander::pander.
Caused by error in `find.package()`:
! there is no package called ‘pander’
ℹ [get_agfd.R:1] @title should be 'Sentence case'
ℹ [get_agfd.R:38] @param should be 'Sentence case'
ℹ [get_agfd.R:45] @param should be 'Sentence case'
ℹ [get_agfd.R:239] @param should be 'Sentence case'
ℹ [get_agfd.R:240] @param should be 'Sentence case'
ℹ [get_agfd.R:251] @title should be 'Sentence case'
ℹ [get_soil_thickness.R:1] @title should be 'Sentence case'
ℹ [get_soil_thickness.R:77] @param should be 'Sentence case'
ℹ [get_soil_thickness.R:129] @param should be 'Sentence case'
ℹ [get_soil_thickness.R:130] @param should be 'Sentence case'
ℹ [inspect_cache.R:1] @title should be 'Sentence case'
ℹ [internal_functions.R:11] @title should be 'Sentence case'
ℹ [internal_functions.R:14] @param should be 'Sentence case'
ℹ [internal_functions.R:33] @param should be 'Sentence case'
ℹ [internal_functions.R:34] @param should be 'Sentence case'
ℹ [internal_functions.R:35] @param should be 'Sentence case'
ℹ [internal_functions.R:37] @param should be 'Sentence case'
ℹ [read.abares-package.R:2] @seealso should be 'Sentence case'
ℹ [read.abares-package.R:2] @seealso should terminate with a full stop, `.`
ℹ [read_aagis_regions.R:1] @title should be 'Sentence case'
ℹ [read_aagis_regions.R:8] @param should be 'Sentence case'
ℹ [read_aagis_regions.R:69] @title should be 'Sentence case'
ℹ [read_aagis_regions.R:75] @param should be 'Sentence case'
ℹ [read_abares_trade.R:1] @title should be 'Sentence case'
ℹ [read_abares_trade.R:12] @param should be 'Sentence case'
ℹ [read_abares_trade.R:48] @title should be 'Sentence case'
ℹ [read_abares_trade.R:52] @param should be 'Sentence case'
ℹ [read_abares_trade_regions.R:1] @title should be 'Sentence case'
ℹ [read_agfd_dt.R:1] @title should be 'Sentence case'
ℹ [read_agfd_dt.R:10] @param should be 'Sentence case'
ℹ [read_agfd_stars.R:1] @title should be 'Sentence case'
ℹ [read_agfd_terra.R:1] @title should be 'Sentence case'
ℹ [read_agfd_tidync.R:1] @title should be 'Sentence case'
ℹ [read_estimates_by_performance_category.R:1] @title should be 'Sentence case'
ℹ [read_estimates_by_size.R:1] @title should be 'Sentence case'
ℹ [read_historical_forecast_database.R:1] @title should be 'Sentence case'
ℹ [read_historical_national_estimates.R:1] @title should be 'Sentence case'
ℹ [read_historical_regional_estimates.R:1] @title should be 'Sentence case'
ℹ [read_historical_state_estimates.R:1] @title should be 'Sentence case'
ℹ [read_soil_thickness_stars.R:1] @title should be 'Sentence case'
ℹ [read_soil_thickness_terra.R:1] @title should be 'Sentence case'
Writing dot-create_soil_thickness_list.Rd
Writing dot-download_soil_thickness.Rd
Writing dot-find_user_cache.Rd
Writing dot-check_class.Rd
Writing dot-retry_download.Rd
Writing dot-check_existing_aagis.Rd
Writing dot-download_aagis_shp.Rd

══ Building ═══════════════════════════════════════════════════════════════════════════════
Setting env vars:
• CFLAGS    : -Wall -pedantic -fdiagnostics-color=always
• CXXFLAGS  : -Wall -pedantic -fdiagnostics-color=always
• CXX11FLAGS: -Wall -pedantic -fdiagnostics-color=always
• CXX14FLAGS: -Wall -pedantic -fdiagnostics-color=always
• CXX17FLAGS: -Wall -pedantic -fdiagnostics-color=always
• CXX20FLAGS: -Wall -pedantic -fdiagnostics-color=always
── R CMD build ────────────────────────────────────────────────────────────────────────────
✔  checking for file 'C:\Data\code\github\rOpenSci\editorials\read.abares/DESCRIPTION' (573ms)
─  preparing 'read.abares': (671ms)
✔  checking DESCRIPTION meta-information ... 
─  installing the package to build vignettes (390ms)
✔  creating vignettes (24.4s)
─  checking for LF line-endings in source and make files and shell scripts (1.2s)
─  checking for empty or unneeded directories
─  building 'read.abares_1.0.0.tar.gz'
   
══ Checking ═══════════════════════════════════════════════════════════════════════════════
Setting env vars:
• _R_CHECK_CRAN_INCOMING_REMOTE_               : FALSE
• _R_CHECK_CRAN_INCOMING_                      : FALSE
• _R_CHECK_FORCE_SUGGESTS_                     : FALSE
• _R_CHECK_PACKAGES_USED_IGNORE_UNUSED_IMPORTS_: FALSE
• NOT_CRAN                                     : true
── R CMD check ────────────────────────────────────────────────────────────────────────────
─  using log directory 'C:/Users/leobr/AppData/Local/Temp/RtmpghvyIL/file14cc2703fd1/read.abares.Rcheck' (429ms)
─  using R version 4.4.3 (2025-02-28 ucrt)
─  using platform: x86_64-w64-mingw32
─  R was compiled by
       gcc.exe (GCC) 13.3.0
       GNU Fortran (GCC) 13.3.0
─  running under: Windows 11 x64 (build 26100)
─  using session charset: UTF-8
─  using options '--no-manual --as-cran'
✔  checking for file 'read.abares/DESCRIPTION' ...
─  checking extension type ... Package
─  this is package 'read.abares' version '1.0.0'
─  package encoding: UTF-8
✔  checking package namespace information ...
N  checking package dependencies (3.5s)
   Packages suggested but not available for checking: 'pander', 'spelling'
✔  checking if this is a source package ... 
✔  checking if there is a namespace
✔  checking for executable files (1.6s)
✔  checking for hidden files and directories ...
✔  checking for portable file names ...
─  checking whether package 'read.abares' can be installed ... [25s] OK (25.3s)
✔  checking installed package size ... 
✔  checking package directory (431ms)
N  checking for future file timestamps ... 
   unable to verify current time
✔  checking 'build' directory
✔  checking DESCRIPTION meta-information (638ms)
N  checking top-level files
   Non-standard file/directory found at top level:
     'air.toml'
✔  checking for left-over files
✔  checking index information (531ms)
✔  checking package subdirectories (2.4s)
✔  checking code files for non-ASCII characters ... 
✔  checking R files for syntax errors ... 
✔  checking whether the package can be loaded (6.1s)
✔  checking whether the package can be loaded with stated dependencies (5.5s)
✔  checking whether the package can be unloaded cleanly (5.9s)
✔  checking whether the namespace can be loaded with stated dependencies (5.9s)
✔  checking whether the namespace can be unloaded cleanly (6.9s)
✔  checking loading without being on the library search path (7.2s)
✔  checking dependencies in R code (13.9s)
✔  checking S3 generic/method consistency (6s)
✔  checking replacement functions (5.8s)
✔  checking foreign function calls (6.3s)
─  checking R code for possible problems ... [33s] OK (33.4s)
✔  checking Rd files (1s)
✔  checking Rd metadata ... 
✔  checking Rd line widths (341ms)
N  checking Rd cross-references (5.5s)
   Package unavailable to check Rd xrefs: 'pander'
✔  checking for missing documentation entries (7.2s)
✔  checking for code/documentation mismatches (18.4s)
✔  checking Rd \usage sections (6.8s)
✔  checking Rd contents (449ms)
✔  checking for unstated dependencies in examples (666ms)
✔  checking installed files from 'inst/doc' (664ms)
✔  checking files in 'vignettes' (661ms)
✔  checking examples (10.7s)
✔  checking for unstated dependencies in 'tests' ... 
─  checking tests (406ms)
─  Running 'spelling.R' (457ms)
   Running the tests in 'tests/testthat.R' failed.
   Last 13 lines of output:
       1. └─read.abares::read_historical_state_estimates() at test-read_historical_state_estimates.R:4:3
       2.   └─read.abares:::.retry_download(...)
       3.     ├─brio::write_file_raw(...)
       4.     ├─httr2::resp_body_raw(...)
       5.     │ └─httr2:::check_response(resp)
       6.     │   └─httr2:::is_response(resp)
       7.     └─httr2::req_perform(...)
       8.       └─base::tryCatch(...)
       9.         └─base (local) tryCatchList(expr, classes, parentenv, handlers)
      10.           └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
      11.             └─value[[3L]](cond)
     
     [ FAIL 11 | WARN 0 | SKIP 0 | PASS 20 ]
     Error: Test failures
     Execution halted
✔  checking for unstated dependencies in vignettes (4.9s)
✔  checking package vignettes ... 
N  checking re-building of vignette outputs (8.1s)
   Note: skipping 'read.abares.Rmd' due to unavailable dependencies: 'pander'
✔  checking for non-standard things in the check directory
✔  checking for detritus in the temp directory ...
   
   See
     'C:/Users/leobr/AppData/Local/Temp/RtmpghvyIL/file14cc2703fd1/read.abares.Rcheck/00check.log'
   for details.
   
   ERROR: Unknown command "TMPDIR=C:/Users/leobr/AppData/Local/Temp/Rtmp6n5EqG/file8945a3e2a0c". Did you mean command "create"?
   Warning in system2("quarto", "-V", stdout = TRUE, env = paste0("TMPDIR=",  :
     running command '"quarto" TMPDIR=C:/Users/leobr/AppData/Local/Temp/Rtmp6n5EqG/file8945a3e2a0c -V' had status 1
   
── R CMD check results ───────────────────────────────────────────── read.abares 1.0.0 ────
Duration: 48m 59.7s

❯ checking tests ...
  See below...

❯ checking package dependencies ... NOTE
  Packages suggested but not available for checking: 'pander', 'spelling'

❯ checking for future file timestamps ... NOTE
  unable to verify current time

❯ checking top-level files ... NOTE
  Non-standard file/directory found at top level:
    'air.toml'

❯ checking Rd cross-references ... NOTE
  Package unavailable to check Rd xrefs: 'pander'

❯ checking re-building of vignette outputs ... NOTE
  Note: skipping 'read.abares.Rmd' due to unavailable dependencies: 'pander'

── Test failures ──────────────────────────────────────────────────────────── testthat ────

> # This file is part of the standard setup for testthat.
> # It is recommended that you do not modify it.
> #
> # Where should you do additional test configuration?
> # Learn more about the roles of various files in:
> # * https://r-pkgs.org/testing-design.html#sec-tests-files-overview
> # * https://testthat.r-lib.org/articles/special-files.html
> 
> library(testthat)
> library(read.abares)

Attaching package: 'read.abares'

The following object is masked from 'package:graphics':

    plot

The following object is masked from 'package:base':

    plot

> 
> test_check("read.abares")
Starting 2 test processes
[ FAIL 11 | WARN 0 | SKIP 0 | PASS 20 ]

══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test-read_estimates_by_performance_category.R:4:3'): read_estimates_by_performance_category() works ──
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Operation too slow. Less than 1 bytes/sec transferred the last 600 seconds
Backtrace:
     ▆
  1. └─read.abares::read_estimates_by_performance_category() at test-read_estimates_by_performance_category.R:4:3
  2.   └─read.abares:::.retry_download(...)
  3.     ├─brio::write_file_raw(...)
  4.     ├─httr2::resp_body_raw(...)
  5.     │ └─httr2:::check_response(resp)
  6.     │   └─httr2:::is_response(resp)
  7.     └─httr2::req_perform(...)
  8.       └─base::tryCatch(...)
  9.         └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 10.           └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 11.             └─value[[3L]](cond)
── Error ('test-read_estimates_by_size.R:4:3'): read_estimates_by_size works ───
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Send failure: Connection was aborted
Backtrace:
     ▆
  1. └─read.abares::read_estimates_by_size() at test-read_estimates_by_size.R:4:3
  2.   └─read.abares:::.retry_download(...)
  3.     ├─brio::write_file_raw(...)
  4.     ├─httr2::resp_body_raw(...)
  5.     │ └─httr2:::check_response(resp)
  6.     │   └─httr2:::is_response(resp)
  7.     └─httr2::req_perform(...)
  8.       └─base::tryCatch(...)
  9.         └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 10.           └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 11.             └─value[[3L]](cond)
── Error ('test-read_aagis_regions.R:8:3'): read_aagis_regions doesn't cache ───
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Operation too slow. Less than 1 bytes/sec transferred the last 600 seconds
Backtrace:
     ▆
  1. └─read.abares::read_aagis_regions(cache = FALSE) at test-read_aagis_regions.R:8:3
  2.   └─read.abares:::.download_aagis_shp(cache)
  3.     └─read.abares:::.retry_download(...)
  4.       ├─brio::write_file_raw(...)
  5.       ├─httr2::resp_body_raw(...)
  6.       │ └─httr2:::check_response(resp)
  7.       │   └─httr2:::is_response(resp)
  8.       └─httr2::req_perform(...)
  9.         └─base::tryCatch(...)
 10.           └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 11.             └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 12.               └─value[[3L]](cond)
── Failure ('test-read_aagis_regions.R:20:3'): read_aagis_regions skips downloading if still in tempdir() ──
`x` is not an S3 object
── Error ('test-read_aagis_regions.R:28:3'): read_aagis_regions caches ─────────
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Operation too slow. Less than 1 bytes/sec transferred the last 600 seconds
Backtrace:
     ▆
  1. └─read.abares::read_aagis_regions(cache = TRUE) at test-read_aagis_regions.R:28:3
  2.   └─read.abares:::.download_aagis_shp(cache)
  3.     └─read.abares:::.retry_download(...)
  4.       ├─brio::write_file_raw(...)
  5.       ├─httr2::resp_body_raw(...)
  6.       │ └─httr2:::check_response(resp)
  7.       │   └─httr2:::is_response(resp)
  8.       └─httr2::req_perform(...)
  9.         └─base::tryCatch(...)
 10.           └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 11.             └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 12.               └─value[[3L]](cond)
── Failure ('test-read_aagis_regions.R:46:3'): read_aagis_regions skips downloading if cache is available ──
`x` is not an S3 object
── Error ('test-read_aagis_regions.R:56:3'): read_aagis_regions does cache ─────
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Operation too slow. Less than 1 bytes/sec transferred the last 600 seconds
Backtrace:
     ▆
  1. └─read.abares::read_aagis_regions(cache = TRUE) at test-read_aagis_regions.R:56:3
  2.   └─read.abares:::.download_aagis_shp(cache)
  3.     └─read.abares:::.retry_download(...)
  4.       ├─brio::write_file_raw(...)
  5.       ├─httr2::resp_body_raw(...)
  6.       │ └─httr2:::check_response(resp)
  7.       │   └─httr2:::is_response(resp)
  8.       └─httr2::req_perform(...)
  9.         └─base::tryCatch(...)
 10.           └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 11.             └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 12.               └─value[[3L]](cond)
── Failure ('test-read_aagis_regions.R:73:3'): read_aagis_regions skips downloading if still in tempdir() ──
`x` is not an S3 object
── Error ('test-read_historical_national_estimates.R:4:3'): read_historical_national_estimates works ──
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Operation too slow. Less than 1 bytes/sec transferred the last 600 seconds
Backtrace:
     ▆
  1. └─read.abares::read_historical_national_estimates() at test-read_historical_national_estimates.R:4:3
  2.   └─read.abares:::.retry_download(...)
  3.     ├─brio::write_file_raw(...)
  4.     ├─httr2::resp_body_raw(...)
  5.     │ └─httr2:::check_response(resp)
  6.     │   └─httr2:::is_response(resp)
  7.     └─httr2::req_perform(...)
  8.       └─base::tryCatch(...)
  9.         └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 10.           └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 11.             └─value[[3L]](cond)
── Error ('test-read_historical_regional_estimates.R:4:3'): read_historical_regional_estimates works ──
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Operation too slow. Less than 1 bytes/sec transferred the last 600 seconds
Backtrace:
     ▆
  1. └─read.abares::read_historical_regional_estimates() at test-read_historical_regional_estimates.R:4:3
  2.   └─read.abares:::.retry_download(...)
  3.     ├─brio::write_file_raw(...)
  4.     ├─httr2::resp_body_raw(...)
  5.     │ └─httr2:::check_response(resp)
  6.     │   └─httr2:::is_response(resp)
  7.     └─httr2::req_perform(...)
  8.       └─base::tryCatch(...)
  9.         └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 10.           └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 11.             └─value[[3L]](cond)
── Error ('test-read_historical_state_estimates.R:4:3'): read_historical_state_estimates works ──
<httr2_failure/httr2_error/rlang_error/error/condition>
Error in `httr2::req_perform(httr2::req_progress(httr2::req_cache(httr2::req_retry(httr2::req_options(httr2::req_user_agent(httr2::request(base_url = url), 
    "read.abares"), http_version = 2, timeout = 2000L), max_tries = .max_tries), 
    path = tempdir())))`: Failed to perform HTTP request.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [www.agriculture.gov.au]: Send failure: Connection was aborted
Backtrace:
     ▆
  1. └─read.abares::read_historical_state_estimates() at test-read_historical_state_estimates.R:4:3
  2.   └─read.abares:::.retry_download(...)
  3.     ├─brio::write_file_raw(...)
  4.     ├─httr2::resp_body_raw(...)
  5.     │ └─httr2:::check_response(resp)
  6.     │   └─httr2:::is_response(resp)
  7.     └─httr2::req_perform(...)
  8.       └─base::tryCatch(...)
  9.         └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 10.           └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 11.             └─value[[3L]](cond)

[ FAIL 11 | WARN 0 | SKIP 0 | PASS 20 ]
Error: Test failures
Execution halted

1 error ✖ | 0 warnings ✔ | 5 notes ✖

@adamhsparks
Copy link
Member Author

@obrl-soil, I'm at a loss here. I don't know what to actually do here, I'm open to suggestions since I'm not able to reproduce this except on GitHub CI. I thought that was just the way GitHub had stuff configured. I don't know how to do any troubleshooting with this. I thought I'd dealt with the flakey connection and slow responses, which in all my testing I had and trying on a Windows machine (I use Mac exclusively) allowed me to rule out a platform issue. So I'm unsure where to look if it's not a curl lib issue on Windows, my first thought when you raised the issue here, I'm not sure what it is. I've never seen anything like it.

@maurolepore, do you have any ideas what we can do here?

@mpadge
Copy link
Member

mpadge commented Mar 6, 2025

@adamhsparks FYI I'd suggest the key seems to be the line:

! Timeout was reached [www.agriculture.gov.au]: Operation too slow. Less than 1 bytes/sec transferred the last 600 seconds

This is similar to behaviour that can happen with the server used in the {osmdata} package, which has very clever throttling mechanisms dependent on a host of factors. Maybe the API you're calling throttles requests from GitHub IP addresses? In which case, you won't be able to do anything about it except re-design tests to avoid downloading ... again 😱 )

@adamhsparks
Copy link
Member Author

adamhsparks commented Mar 6, 2025

@mpadge, I've given up with trying to DL from GitHub, this is a problem for @obrl-soil who can download the files with her browser but not the package, I'm stumped.

But if we can fix it for Lauren, perhaps we can fix it for GitHub too.

@mpadge
Copy link
Member

mpadge commented Mar 6, 2025

Browser has a different user agent. Maybe they're even throttling everything that is not a standard browser user agent? You can fake user agents in the package to pretend it's a browser, but that just starts a game of cat-and-mouse.

@adamhsparks
Copy link
Member Author

adamhsparks commented Mar 6, 2025

I thought about that, and it works on every computer I’ve tried with the default user agent for {httr2}.

@obrl-soil
Copy link

I had previously tried messing around with the .retry_download() function because I wondered whether the use of piped code was causing a problem (no) and as part of that I did try skipping the user agent at one point. Didn't help.

I also tried the SSL mod described here, no change.

Is it possible this is a server-specific issue? @adamhsparks do you have any contacts at ABARES' IT dept?

@adamhsparks
Copy link
Member Author

I don’t know anyone at ABARES, no.

I’m baffled. It works in a browser but suddenly not here and no real error either. I wish I could recreate this. Maybe using curl from the CLI would be insightful. I can try that tomorrow and see what happens if I’m not using R, I guess.

@obrl-soil
Copy link

This is definitely domain-specific. I've tested every URL stored in your package. The ones linking to the ABARES publications library at https://daff.ent.sirsidynix.net.au are fine, as is the link to their AWS-hosted soil thickness data. Only the data stored in https://www.agriculture.gov.au/sites/default/files/documents/ is a problem. FWIW I also have no problems downloading from e.g. data.gov.au or various other state-based portals.

I'm of the opinion that the datasets stored on the website should be moved into the ABARES publications library or onto data.gov.au as a matter of best practice (FAIR principles, proper metadata management etc) but you'd have to take that up with the site admins. If they don't action your request, you may be down to one or both of the following options:

  • re-host copies of the affected files yourself, and take on the maintenance burden of keeping your version of the data up to date
  • Change the workflow for the affected datasets to handle the issue. E.g. if the download times out, have a message print to the console describing the problem and directing the user to download the data manually. read.abares could then be used to do the data cleaning steps you've built in to the package. Less maintenance for you but more overhead for the user.

Not ideal, I know :/

@adamhsparks
Copy link
Member Author

Not ideal, but useful info. Let me see what I can find.

@adamhsparks
Copy link
Member Author

adamhsparks commented Mar 7, 2025

@obrl-soil, can you use curl and give me the output of,
bash curl -vO https://www.agriculture.gov.au/sites/default/files/documents/fdp-national-historical.csv?

This is what I see on my machine.

~/tmp via  v3.4.1
♥ ❯ curl -vO https://www.agriculture.gov.au/sites/default/files/documents/fdp-national-historical.csv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host www.agriculture.gov.au:443 was resolved.
* IPv6: 2600:1415:11::1737:f29a, 2600:1415:11::1737:f283
* IPv4: 184.84.150.48, 184.84.150.145
*   Trying 184.84.150.48:443...
* Connected to www.agriculture.gov.au (184.84.150.48) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [327 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [29 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2616 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=www.agriculture.gov.au
*  start date: Mar  6 10:20:27 2025 GMT
*  expire date: Jun  4 10:20:26 2025 GMT
*  subjectAltName: host "www.agriculture.gov.au" matched cert's "www.agriculture.gov.au"
*  issuer: C=US; O=Let's Encrypt; CN=R11
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.agriculture.gov.au/sites/default/files/documents/fdp-national-historical.csv
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.agriculture.gov.au]
* [HTTP/2] [1] [:path: /sites/default/files/documents/fdp-national-historical.csv]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /sites/default/files/documents/fdp-national-historical.csv HTTP/2
> Host: www.agriculture.gov.au
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< expires: Sat, 08 Mar 2025 00:10:01 GMT
< last-modified: Mon, 02 Dec 2024 23:51:40 GMT
< etag: "674e480c-1742c1"
< cache-control: max-age=1800
< x-frame-options: SAMEORIGIN
< x-robots-tag: all
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000; includeSubDomains
< x-lagoon: amazeeio-govcms5>sigsci-ingress-nginx>daff-master:nginx>
< content-type: application/octet-stream
< date: Fri, 07 Mar 2025 23:41:14 GMT
<
{ [16375 bytes data]
100 1488k    0 1488k    0     0  2585k      0 --:--:-- --:--:-- --:--:-- 2589k
* Connection #0 to host www.agriculture.gov.au left intact

@obrl-soil
Copy link

Sure,

C:\Users\leobr>curl -vO https://www.agriculture.gov.au/sites/default/files/documents/fdp-national-historical.csv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host www.agriculture.gov.au:443 was resolved.
* IPv6: (none)
* IPv4: 131.203.3.10
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 131.203.3.10:443...
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* Connected to www.agriculture.gov.au (131.203.3.10) port 443
* using HTTP/1.x
> GET /sites/default/files/documents/fdp-national-historical.csv HTTP/1.1
> Host: www.agriculture.gov.au
> User-Agent: curl/8.10.1
> Accept: */*
>
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* Request completely sent off
  0     0    0     0    0     0      0      0 --:--:--  0:10:00 --:--:--     0* Recv failure: Connection was reset
* schannel: recv returned CURLE_RECV_ERROR
  0     0    0     0    0     0      0      0 --:--:--  0:10:00 --:--:--     0
* closing connection #0
curl: (56) Recv failure: Connection was reset

@adamhsparks
Copy link
Member Author

thanks, that's something for me to try to use with troubleshooting this.

@adamhsparks
Copy link
Member Author

adamhsparks commented Mar 9, 2025

Ok, this clearly is not a {read.abares} package issue given that it fails with curl from the CLI for specific computers. So, based on the logs provided, I have a faint memory of having issues with Windows, {curl}/curl and schannel when writing {weatherOz} and we changed the backend and it fixed our issues (locally). @obrl-soil, can we try changing the curl backend on your machine machine, e.g., https://jeroen.r-universe.dev/articles/curl/windows.html, and see if that fixes the issues we're seeing?

@obrl-soil
Copy link

I already tried that in R without success, and setting a system-wide environment variable didn't change anything. I did notice a potential issue with multiple versions of curl (see your slack thread). Running the following on the command line to specifically invoke the Rtools instance of curl produced another failure, but at least its a new and different one?

c:\Data>C:\rtools44\usr\bin\curl.exe -vO https://www.agriculture.gov.au/sites/default/files/documents/fdp-national-historical.csv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host www.agriculture.gov.au:443 was resolved.
* IPv6: 2404:4408:5:51cc::83cb:30a, 2404:4408:5:51cc::83cb:309
* IPv4: 131.203.3.10, 131.203.3.9
*   Trying [2404:4408:5:51cc::83cb:30a]:443...
* connect to 2404:4408:5:51cc::83cb:30a port 443 from :: port 50129 failed: Network is unreachable
*   Trying [2404:4408:5:51cc::83cb:309]:443...
* connect to 2404:4408:5:51cc::83cb:309 port 443 from :: port 50130 failed: Network is unreachable
*   Trying 131.203.3.10:443...
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /usr/ssl/certs/ca-bundle.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [29 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2616 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=www.agriculture.gov.au
*  start date: Mar  6 10:20:27 2025 GMT
*  expire date: Jun  4 10:20:26 2025 GMT
*  subjectAltName: host "www.agriculture.gov.au" matched cert's "www.agriculture.gov.au"
*  issuer: C=US; O=Let's Encrypt; CN=R11
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to www.agriculture.gov.au (131.203.3.10) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.agriculture.gov.au/sites/default/files/documents/fdp-national-historical.csv
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.agriculture.gov.au]
* [HTTP/2] [1] [:path: /sites/default/files/documents/fdp-national-historical.csv]
* [HTTP/2] [1] [user-agent: curl/8.12.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /sites/default/files/documents/fdp-national-historical.csv HTTP/2
> Host: www.agriculture.gov.au
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
} [5 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host www.agriculture.gov.au left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

@adamhsparks
Copy link
Member Author

Ok, so, I just had a browse over my e-mail notification of discussions on the Posit forum and this gem popped up, https://forum.posit.co/t/httr2-request-fails-ok-in-browser/197965/2, look familiar? I thought so.

So, I've tried the suggested fix therein,

Lines 54:57 of internal_functions.R are now:

 httr2::req_user_agent("Mozilla/5.0") |>
    httr2::req_headers("Accept-Encoding" = "identity") |>
    httr2::req_headers("Connection" = "Keep-Alive") |>
    httr2::req_options(http_version = 2, timeout = 2000L) |>

Keeping the http_version = 2, timeout = 2000L) that I already had in place. This again worksforme. 🙃

@obrl-soil
Copy link

obrl-soil commented Mar 11, 2025

It finally worked!!! I did have to remove that openssl variable though

It also seems fine if I reset the user agent to "read.abares", so I'd say its the header commands that matter

@adamhsparks
Copy link
Member Author

FINALLY!!! I'll clean up and we'll go with it.

@obrl-soil
Copy link

Hey - just trying to finalise my editor comments. Package check functions are working fine but I'm running into odd hiccups using the package interactively. For instance, running the example code from read_soil_thickness_terra():

x <- get_soil_thickness(cache = TRUE) |>
+   read_soil_thickness_terra()
Error in list_files(file, ignore_missing, TRUE, verbosity) : 
  File '' does not exist.
In addition: Warning message:
In file.rename(file.path(download_dir, "staiar9cl__05911a01eg_geo___/"),  :
  cannot rename file 'C:/Users/leobr/AppData/Local/R/cache/R/read.abares/staiar9cl__05911a01eg_geo___' to 'C:/Users/leobr/AppData/Local/R/cache/R/read.abares/soil_thickness_dir', reason 'Access is denied'

Not sure why this is happening. The files have downloaded and nothing is tagged as read-only in that folder.

Also, for

agfd <- get_agfd(cache = TRUE) |>
+   read_agfd_terra()
Error in `purrr::map()`:                              
ℹ In index: 1.
Caused by error:
! [rast] file does not exist: read.abares.agfd.nc.files
Run `rlang::last_trace()` to see where the error occurred.
Warning message:
read.abares.agfd.nc.files: No such file or directory (GDAL error 4) 

The data is downloaded and present in the cache, but I don't think get_agfd() is listing the files correctly in its output object.

Can you replicate these issues?

@adamhsparks
Copy link
Member Author

Yes, and I found where the issue is. Now I need to understand why.

Sorry about this, I thought it was all in good order.

@adamhsparks
Copy link
Member Author

Ok! @obrl-soil, I'm very sorry about that. I've now inspected, detected, debugged and refactored and it's now passing all tests and has improved code-coverage, 98%, along with better guard rails.

I hope it's good to go now. This has been a problematic package, to say the least and it shouldn't be that complicated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants