Optionally use pyarrow types in to_geodataframe #31
Conversation
This updates to_geodataframe to optionally use pyarrow types, rather than NumPy. These types let us faithfully represent the actual nested types, rather than casting everything to `object`.
TomAugspurger
force-pushed
the
feature/arrow-types
branch
from
73fdfac to
fb798f4
Compare
|
Awesome! Excited to see this! |
stac_geoparquet/stac_geoparquet.py
Outdated
| for k, v in items2.items(): | ||
| if k in DATETIME_COLUMNS: | ||
| items2[k] = pd.arrays.ArrowExtensionArray( | ||
| pa.array(pd.to_datetime(v, format="ISO8601")) |
There was a problem hiding this comment.
I want the output here to be identical to what we're getting in #27.
Right now, the date time columns from this PR end up with nanosecond precision, while Kyle's PR has microsecond precision. I'm not sure if there's a correct default, but we should try and get them the same.
There was a problem hiding this comment.
Rather than guessing, I've made this a parameter for to_geodataframe. The default is ns which will be compatible with what pandas was doing previously for NumPy dtypes.
We're actually still relying on pandas' to_datetime for parsing strings into timestamps, before casting to Arrow. Apparently pyarrow's pc.strptime doesn't support fractional seconds yet: apache/arrow#20146
This updates to_geodataframe to optionally use pyarrow types, rather than NumPy. These types let us faithfully represent the actual nested types, rather than casting everything to
object. I think this will be a good default in the future. For now, it's just optional.There are some changes to the actual values associated with this change, related to how optional fields are stored.
If the source STAC documents had some values like
the new output will have a
structtype with two fieldshrefandtitle. The value ofa.titlewill beNone, instead of just being absent.