Skip to content
Closed
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
6aae933
initial commit
jph6366 Aug 4, 2025
53a4b88
rigorously tested yet no valid gpkg writer just yet.
jph6366 Sep 9, 2025
43a87af
SQLite package install, read/write includes, SQLite.DBInterface
jph6366 Sep 9, 2025
91ed1ef
formatter installed, addressing comments and some varname changes, an…
jph6366 Sep 10, 2025
3166a1c
addressing commit suggestions
jph6366 Sep 11, 2025
0255232
missed this one
jph6366 Sep 11, 2025
5e60488
another mistake missed
jph6366 Sep 11, 2025
39730ce
Reader is sufficiently spec-compliant for further testing, addressing…
jph6366 Sep 12, 2025
26b7602
removed newline
jph6366 Sep 12, 2025
0077fef
removed returns for code style
jph6366 Sep 12, 2025
de56781
creation of wkb.jl, removing/recycling redundant and/or unused code, …
jph6366 Sep 15, 2025
6c3c505
formatting and include(wkb.jl)
jph6366 Sep 15, 2025
5dc18fb
incoporated custom SQLite data types, annotated type for wkbgeometry,…
jph6366 Sep 18, 2025
46b20b9
removed minor version
jph6366 Sep 18, 2025
6538130
passing all tests, added branch for novalues, and added wkblinearring…
jph6366 Sep 19, 2025
e93f91c
missing blob error handling
jph6366 Sep 20, 2025
d73a169
removed unnecessary arrays
jph6366 Sep 20, 2025
1cf927e
added necessary arrays comprehensions back, database needs to be clos…
jph6366 Sep 22, 2025
4201eb2
fixing Ring and Rope expressions to include splat and rearranging SQL…
jph6366 Sep 23, 2025
f401635
add Random, some capi sqlite, sql faster and safer in a transaction, …
jph6366 Sep 25, 2025
82d635a
Merge branch 'master' into gpkg_io
jph6366 Sep 25, 2025
acc948c
removed unnecessary Random and Dates, and reorganized create tables a…
jph6366 Sep 27, 2025
dc722cc
removed id ok autoincrement, undo my evil test changes
jph6366 Sep 27, 2025
f7e4e85
tests passing! simple features supported! need to test for multi feat…
jph6366 Oct 2, 2025
b2451b1
multi. fix, added util to infer Multi const geometry,
jph6366 Oct 2, 2025
31d9cba
Merge branch 'master' into gpkg_io
jph6366 Oct 2, 2025
a884e88
formatting
jph6366 Oct 2, 2025
e9cd713
Merge branch 'gpkg_io' of https://github.com/jph6366/GeoIO.jl into gp…
jph6366 Oct 2, 2025
6d2244c
mit license header
jph6366 Oct 2, 2025
7cd4851
resolving reviewed changes
jph6366 Oct 7, 2025
f98d6b6
optimized meshestowkb{T<:WKBMulti}
jph6366 Oct 8, 2025
ddd08d5
simplified wkb reader/writer, added docstrings
jph6366 Oct 9, 2025
b058783
evil test changes to implement AUTOINCREMENT
jph6366 Oct 9, 2025
347fe4b
Merge branch 'gpkg_io' of https://github.com/jph6366/GeoIO.jl into gp…
jph6366 Oct 9, 2025
4de2764
minor fixes
jph6366 Oct 9, 2025
97a03e2
added spatial r-tree indexes and gpkg_extensions, also fix for writin…
jph6366 Oct 10, 2025
f98563c
enums are best for this usecase
jph6366 Oct 10, 2025
9db3590
Just comments
jph6366 Oct 10, 2025
9d6574f
adding suggest changes and slightly refactored gpkgvalues
jph6366 Oct 16, 2025
0a12360
another round of resolves
jph6366 Oct 23, 2025
41e73d2
Apply suggestion from @juliohm
juliohm Oct 24, 2025
3e4b02e
Apply suggestion from @juliohm
juliohm Oct 24, 2025
0ee3192
Merge branch 'JuliaEarth:master' into gpkg_io
jph6366 Oct 24, 2025
8ad4c2a
remove evil and purge smelly code
jph6366 Oct 24, 2025
c1ec60d
Merge branch 'gpkg_io' of https://github.com/jph6366/GeoIO.jl into gp…
jph6366 Oct 24, 2025
c0c06a4
revert test changes
jph6366 Oct 24, 2025
ff406c0
gpkgread refactor; more comments
jph6366 Oct 29, 2025
ec9de5a
write.jl comments; wkb.jl remains for review
jph6366 Oct 30, 2025
c23fdb2
wkb comments
jph6366 Nov 3, 2025
59428a4
removed isequal
jph6366 Nov 3, 2025
62c9393
GeoIO.load changes only
jph6366 Nov 3, 2025
e4a40f0
Remove inclusion of write.jl in gpkg.jl
jph6366 Nov 3, 2025
a83e48b
fixed tests; ArchGDAL AUTOINCREMENT issue
jph6366 Nov 3, 2025
7e43151
primary key one-indexed in PRAGMA tableinfo
jph6366 Nov 4, 2025
ed4bb0c
Merge branch 'master' of https://github.com/jph6366/GeoIO.jl into gpk…
jph6366 Nov 4, 2025
57f3a66
update dependency versions
jph6366 Nov 4, 2025
72335cd
removed consts used for writer
jph6366 Nov 4, 2025
97ca89f
trimming fat; refactor wkb; clean comments
jph6366 Nov 7, 2025
694853a
mv Multi(geoms); simpler wkb2geom; test GeoArtifacst
jph6366 Nov 7, 2025
b99a299
commit suggestions; modular gpkg utils; refactor varnames; handle mut…
jph6366 Nov 8, 2025
c47aeeb
remove return
jph6366 Nov 8, 2025
1ad329c
better multi-layer behavior
jph6366 Nov 9, 2025
db61a1c
idioms
jph6366 Nov 9, 2025
00e988d
More adjustments
juliohm Nov 9, 2025
00a5290
k-th layers, hoist IOBlubber
jph6366 Nov 9, 2025
fb6ef04
More adjustments
juliohm Nov 9, 2025
79560a3
More adjustments
juliohm Nov 9, 2025
f3dce85
More adjustments
juliohm Nov 9, 2025
12a2597
More adjustments
juliohm Nov 9, 2025
76193b6
More adjustments
juliohm Nov 9, 2025
6aba388
More adjustments
juliohm Nov 9, 2025
3dd8bda
hotfixes and revert adjustment
jph6366 Nov 9, 2025
d604a94
Merge branch 'gpkg_io' of https://github.com/jph6366/GeoIO.jl into gp…
jph6366 Nov 9, 2025
c563881
Simplify CRS logic
juliohm Nov 10, 2025
0b52233
featuretable varname; magic gp.
jph6366 Nov 10, 2025
ed7ac0c
redo adjustment; simplfiy query; remove unneccessary param.
jph6366 Nov 10, 2025
99a37da
More adjustments
juliohm Nov 11, 2025
435f6bb
More adjusments
juliohm Nov 11, 2025
7751cd1
More adjustments
juliohm Nov 11, 2025
c1a2ea7
More adjustments
juliohm Nov 11, 2025
7399fa2
simply select stmt; zextent refactor
jph6366 Nov 11, 2025
ef477bb
Refactor wkb2geom
juliohm Nov 13, 2025
b912fa8
More adjustments
juliohm Nov 13, 2025
2aa6ce0
More adjustments
juliohm Nov 13, 2025
5efb822
More adjustments
juliohm Nov 13, 2025
f0ff9b2
wkb refactors
jph6366 Nov 13, 2025
2a012bc
simplify and rewrite headerskip
jph6366 Nov 16, 2025
bd2a0ed
batch wkb2points
jph6366 Nov 18, 2025
b154726
refactor wkbtype z mask
jph6366 Nov 18, 2025
f2fcb78
tested on 3dLineString
jph6366 Nov 18, 2025
0ead8f2
raw(::LatLonAlt)
jph6366 Nov 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
Colors = "5ae59095-9a9b-59fe-a467-6f913c188581"
CommonDataModel = "1fbeeb36-5f17-413c-809b-666fb144f157"
CoordRefSystems = "b46f11dc-f210-4604-bfba-323c1ec968cb"
DBInterface = "a10d1c49-ce27-4219-8d33-6db1a4562965"
FileIO = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549"
Format = "1fa38f19-a742-5d3f-a2b9-30dd87b9d5f8"
GRIBDatasets = "82be9cdb-ee19-4151-bdb3-b400788d9abc"
Expand All @@ -27,6 +28,7 @@ PlyIO = "42171d58-473b-503a-8d5f-782019eb09ec"
PrecompileTools = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
PrettyTables = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
ReadVTK = "dc215faf-f008-4882-a9f7-a79a826fadc3"
SQLite = "0aa819cd-b072-5ff4-a722-6bc24af294d9"
Shapefile = "8e980c4a-a4fe-5da2-b3a7-4b4b0353a2f4"
StaticArrays = "90137ffa-7385-5640-81b9-e52037218182"
Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
Expand All @@ -41,6 +43,7 @@ CSV = "0.10"
Colors = "0.12, 0.13"
CommonDataModel = "0.2, 0.3"
CoordRefSystems = "0.18"
DBInterface = "2.6"
FileIO = "1.16"
Format = "1.3"
GRIBDatasets = "0.3, 0.4"
Expand All @@ -59,6 +62,7 @@ PlyIO = "1.1"
PrecompileTools = "1.2"
PrettyTables = "2.2"
ReadVTK = "0.2"
SQLite = "1.6"
Shapefile = "0.13"
StaticArrays = "1.6"
Tables = "1.7"
Expand Down
7 changes: 6 additions & 1 deletion src/GeoIO.jl
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ import PlyIO
# CSV format
import CSV

# Database interfaces
import DBInterface
import SQLite
Comment on lines +41 to +43
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need both dependencies? Could you please elaborate on the rationale for using SQLite.jl and DBInterface.jl in different places of the code? Convenience? DBInterface.jl provides higher-level constructs that justify its use?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this is a low-level method that just executes the SQL statement,
but does not retrieve any data from db.
To get the results of a SQL query, it is recommended to use DBInterface.execute.

SQLite.execute(db, stmt) executes but returns nothing

Comment on lines +42 to +43
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asked a question that was not answered about the need of importing these two.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left this as my answer to the previous comment

This is taken from the docstring for SQLite.execute

Note: this is a low-level method that just executes the SQL statement,
but does not retrieve any data from db.
To get the results of a SQL query, it is recommended to use DBInterface.execute.

SQLite.execute(db, stmt) executes but returns nothing

...

DBInterface.execute is the only we can retrieve/read anything from Querys


# geostats formats
import GslibIO

Expand Down Expand Up @@ -72,7 +76,7 @@ const CDMEXTS = [".grib", ".nc"]
const FORMATS = [
(extension=".csv", load="CSV.jl", save="CSV.jl"),
(extension=".geojson", load="GeoJSON.jl", save="GeoJSON.jl"),
(extension=".gpkg", load="ArchGDAL.jl", save="ArchGDAL.jl"),
(extension=".gpkg", load="GeoIO.jl", save="GeoIO.jl"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(extension=".gpkg", load="GeoIO.jl", save="GeoIO.jl"),
(extension=".gpkg", load="GeoIO.jl", save="ArchGDAL.jl"),

(extension=".grib", load="GRIBDatasets.jl", save=""),
(extension=".gslib", load="GslibIO.jl", save="GslibIO.jl"),
(extension=".jpeg", load="ImageIO.jl", save="ImageIO.jl"),
Expand Down Expand Up @@ -127,6 +131,7 @@ include("extra/csv.jl")
include("extra/gdal.jl")
include("extra/geotiff.jl")
include("extra/gis.jl")
include("extra/gpkg.jl")
include("extra/img.jl")
include("extra/msh.jl")
include("extra/obj.jl")
Expand Down
4 changes: 4 additions & 0 deletions src/extra/gis.jl
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ function giswrite(fname, geotable; warn, kwargs...)
elseif endswith(fname, ".parquet")
CRS = crs(domain(geotable))
GPQ.write(fname, geotable, (:geometry,), projjson(CRS); kwargs...)
elseif endswith(fname, ".gpkg")
gpkgwrite(fname, geotable; kwargs...)
else # fallback to GDAL
agwrite(fname, geotable; kwargs...)
end
Expand All @@ -63,6 +65,8 @@ function gistable(fname; layer, numtype, kwargs...)
return GJS.read(fname; numbertype=numtype, kwargs...)
elseif endswith(fname, ".parquet")
return GPQ.read(fname; kwargs...)
elseif endswith(fname, ".gpkg")
return gpkgread(fname; layer, kwargs...)
else # fallback to GDAL
data = AG.read(fname; kwargs...)
return AG.getlayer(data, layer - 1)
Expand Down
101 changes: 101 additions & 0 deletions src/extra/gpkg.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# ------------------------------------------------------------------
# Licensed under the MIT License. See LICENSE in the project root.
# ------------------------------------------------------------------

# According to https://www.geopackage.org/spec/#r2
# a GeoPackage should contain "GPKG" in ASCII in
# "application_id" field of SQLite db header
const GP10_APPLICATION_ID = Int(0x47503130)
const GP11_APPLICATION_ID = Int(0x47503131)
const GPKG_APPLICATION_ID = Int(0x47504B47)
const GPKG_1_2_VERSION = 10200
const GPKG_1_3_VERSION = 10300
const GPKG_1_4_VERSION = 10400

# types used within the GeoPackageBinary SQL BLOBs
@enum wkbGeometryType begin
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move these types to a separate wkb.jl file?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, planning on coding up a wkb.jl prototype (hopefully) later today.

Copy link
Author

@jph6366 jph6366 Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a preamble of wkb.jl started, working on optimizing and testing throughout today.

edit (Sunday Sept 14, 2025):

Was visiting family this weekend. Back to working this out today.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Next commit (in maybe a few days) will contain some code for struct GeoPackageGeometry and function SQLite.sqlitetype_(::Type{GeoPackageGeometry}) to idiomatically define GeoPackage SQL geometry types. The different geometry type methods will be defined by overloading methods 'meshfromwkb' & 'writewkbgeom' and dispatched in a similar fashion to CoordRefSystems crs codes. Planning to progress towards optimized julia code for type stable function calls and also avoiding unnecessary allocations, also to divide and conquer WKB logic to its own file (wkb.jl). Ideally this implementation can be reused to support future WKB Geometry integrations into GeoStats.jl and offer an alternative to WellKnownGeometry.jl's implementation of GeoInterface.

This is my beginnings of developing new wkb.jl

abstract type WKBFlavour end
abstract type WKB{Code} <: WKBFlavour end
abstract type EXT{Code} <: WKBFlavour end
abstract type ISO{Code} <: WKBFlavour end

struct WKBArray{N}
    wkbtype::UInt32
    wkbgeom::SVector{N, UInt8}
end

function wkbarray(; wkbtype = UInt32(0), wkbgeom = SVector{0, UInt8}())
    WKBArray{length(wkbgeom)}(wkbtype, SVector{length(wkbgeom), UInt8}(wkbgeom))
end

function getwkb(code::Type{WKBFlavour})
  throw(ArgumentError("""
  The provided code $code is not mapped to a WKB Geometry type yet.
  """))
end

macro wkballoc(WKBCode, WKBArray)
    expr = quote
        getwkb(::Type{$WKBCode}) = $WKBArray;
        wkbspec(::Type{$WKBArray}) = $WKBCode;
    end
    esc(expr)
end

@wkballoc WKB{1} wkbarray(; wkbtype= UInt32(1), wkbgeom = @SVector zeros(UInt8, 21))

I'm still fairly new to metaprogramming in Julia and I am confident that I need some guidance on how to navigate this crude wkb.jl prototype as well as the use of StaticArrays.

Also still need to incorporate custom SQL datatype
function SQLite.sqlitetype_(::Type{WKBGeometry})

I am continuing my work on this today.

Update: 09/18/2025

function SQLite.sqlitetype_(WKBGeometry) has been incorporated...

I am digging in tonight to make proper use and optimized code of wkb.jl and ideally getting rid of the enum wkbGeometryType

Update: 09/22/2025

Warmup

  • WKBPoint -> WKBLinestring -> WKBPolygon
    • SVector
    • Macro
      • WKBCode & WKBFlavor
      • preallocated WKBGeometryType <: GenericWKBArray
      • ....
    • ? Broacasting ( bswap, crs )?

wkbUnknown = 0
wkbPoint = 1
wkbLineString = 2
wkbPolygon = 3
wkbMultiPoint = 4
wkbMultiLineString = 5
wkbMultiPolygon = 6
wkbGeometryCollection = 7
wkbCircularString = 8
wkbCompoundCurve = 9
wkbCurvePolygon = 10
wkbMultiCurve = 11
wkbMultiSurface = 12
wkbCurve = 13
wkbSurface = 14

wkbNone = 100 # pure attribute records
wkbLinearRing = 101

# ISO SQL/MM Part 3: Spatial
# Z-aware types
wkbPointZ = 1001
wkbLineStringZ = 1002
wkbPolygonZ = 1003
wkbMultiPointZ = 1004
wkbMultiLineStringZ = 1005
wkbMultiPolygonZ = 1006
wkbGeometryCollectionZ = 1007
wkbCircularStringZ = 1008
wkbCompoundCurveZ = 1009
wkbCurvePolygonZ = 1010
wkbMultiCurveZ = 1011
wkbMultiSurfaceZ = 1012
wkbCurveZ = 1013
wkbSurfaceZ = 1014

# ISO SQL/MM Part 3.
# M-aware types
wkbPointM = 2001
wkbLineStringM = 2002
wkbPolygonM = 2003
wkbMultiPointM = 2004
wkbMultiLineStringM = 2005
wkbMultiPolygonM = 2006
wkbGeometryCollectionM = 2007
wkbCircularStringM = 2008
wkbCompoundCurveM = 2009
wkbCurvePolygonM = 2010
wkbMultiCurveM = 2011
wkbMultiSurfaceM = 2012
wkbCurveM = 2013
wkbSurfaceM = 2014

# ISO SQL/MM Part 3.
# ZM-aware types ... Meshes.jl doesn't generally support this?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These ZM types are a really weird convention. I understand that Z refers to the 3rd coordinate in a XYZ Cartesian coordinate reference system and that M refers to a generic measurement stored along the coordinates.

Meshes.jl treats coordinates as first-class citizens without all these hacks to accommodate 2D, 2.5D and 3D applications. We only have Cartesian coordinates, which are n-dimensional. In practice they are either 2D with x and y fields or 3D with x, y and z fields.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add the enums for processing the input data correctly and for throwing errors or warnings whenever the M coordinate is present. If the Z coordinate is present, we just need to construct Meshes.jl geometries with 3D Cartesian coordinates.

wkbPointZM = 3001
wkbLineStringZM = 3002
wkbPolygonZM = 3003
wkbMultiPointZM = 3004
wkbMultiLineStringZM = 3005
wkbMultiPolygonZM = 3006
wkbGeometryCollectionZM = 3007
wkbCircularStringZM = 3008
wkbCompoundCurveZM = 3009
wkbCurvePolygonZM = 3010
wkbMultiCurveZM = 3011
wkbMultiSurfaceZM = 3012
wkbCurveZM = 3013
wkbSurfaceZM = 3014

# 2.5D extension as per 99-402
# https://lists.osgeo.org/pipermail/postgis-devel/2004-December/000702.html

# Julia throws InexactError: convert(Int32, 0x8000000x)
wkbPoint25D = -2147483647 # 0x80000001
wkbLineString25D = -2147483646 # 0x80000002
wkbPolygon25D = -2147483645 # 0x80000003
wkbMultiPoint25D = -2147483644 # 0x80000004
wkbMultiLineString25D = -2147483643 # 0x80000005
wkbMultiPolygon25D = -2147483642 # 0x80000006
wkbGeometryCollection25D = -2147483641 # 0x80000007
end

include("gpkg/read.jl")
include("gpkg/write.jl")
Loading
Loading