[WIP] Las 1.3 and 1.4 read support. #16

evetion · 2018-04-01T08:36:23Z

Read in EVLR
Support custom attributes pasted on top of standard point types 😞
Read in waveform data
Tests for waveform

evetion · 2018-04-01T08:51:42Z

@visr In my opinion this requires us to change our header, points = load() ways, because of several reasons:

EVLR can only be read after the pointdata (if we support pipes)
Waveform data in the las file (or externally) doesnt fit in either header or points
1.4 supports very large files (both in counts as in the extra waveform data) which leads to not wanting to load all points or waveform at once.

I'm thinking about the following API (apart from the whole datastreams).

header = load(file)
points = points(header)
waves = waves(header)

Where the header (could be renamed) stores the file/stream reference and can load the points or waves on the fly. All small VLRs can be stored in the header, but larger ones (internal waves) will only be streamed/mmapped.

I'm also thinking of disabling pipes for these versions and updating LazIO with 1.3/1.4 support. Pipes for loading .laz files can't work with this API at all and with the filesizes it quickly leads to memory problems.

Let me know your thoughts.

evetion · 2018-04-01T09:01:17Z

I'm aware of the las14 branch started by @c42f, and will probably merge the work in there (mainly fixed size strings) at a later stage.

visr · 2018-04-01T09:29:37Z

Exciting! Yeah I realize the API will probably need to change to support this. Let's try to get it right this time. Will need some time to go over this.

Regarding the fixed size strings, I was also looking at https://github.com/JuliaComputing/FixedSizeStrings.jl, but it's not very active, so might be better to provide for that in here for now.

evetion · 2018-04-17T19:51:29Z

Got a first working version of waveform parsing for the internal 1.3 spec. Still using hardcoded vlrs and assuming 8 bits per sample, but we're getting there.

h, p, e = load("test/LDR100301_120340_4.LAS")
wf = LasIO.waveform(p[3], h, e[end])
plot(wf[:, 1], wf[:,end], label="$(p[3])", xlabel="Distance (ps)", ylabel="Sample (V)", title="Digitized waveform")

codecov-io · 2018-04-22T12:27:19Z

Codecov Report

Merging #16 into master will decrease coverage by 5.86%.
The diff coverage is 43.68%.

@@            Coverage Diff             @@
##           master      #16      +/-   ##
==========================================
- Coverage   62.62%   56.75%   -5.87%     
==========================================
  Files           8       10       +2     
  Lines         396      481      +85     
==========================================
+ Hits          248      273      +25     
- Misses        148      208      +60

Impacted Files	Coverage Δ
src/LasIO.jl	`100% <ø> (ø)`	⬆️
src/waveform.jl	`0% <0%> (ø)`
src/point.jl	`61.53% <0%> (-2.47%)`	⬇️
src/meta.jl	`3.12% <0%> (-1.43%)`	⬇️
src/srs.jl	`91.54% <100%> (-0.02%)`	⬇️
src/vlrs.jl	`39.72% <29.03%> (-60.28%)`	⬇️
src/fileio.jl	`43.75% <42.3%> (-2.41%)`	⬇️
src/header.jl	`87.37% <75%> (+19.7%)`	⬆️
src/fixedstrings.jl	`95% <95%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c35f4c...2e87503. Read the comment docs.

evetion · 2018-04-24T11:54:29Z

@visr Could you review the current functionality and the way it's implemented? The goal here is to fully support reading LAS 1.x.

I will start writing tests and documentation now. Write support can be added later in another PR.

visr

I'm also thinking of disabling pipes for these versions and updating LazIO with 1.3/1.4 support. Pipes for loading .laz files can't work with this API at all and with the filesizes it quickly leads to memory problems.

Yeah dropping this piping LAZ support is fine by me. We can work on getting LazIO up to speed afterwards.

I'm thinking about the following API (apart from the whole datastreams).
header = load(file)
points = points(header)
waves = waves(header)
Where the header (could be renamed) stores the file/stream reference and can load the points or waves on the fly. All small VLRs can be stored in the header, but larger ones (internal waves) will only be streamed/mmapped.

Something like that sounds good to me, it's also in line with #4 (comment). How do you decide what's a small VLR and what is big though? Do you mean VLR vs EVLR? We don't want to have a situation where people don't have enough RAM to read the header since it also reads large VLRs. So perhaps we should take it out of the header altogether? It aligns also better with the structure described in the spec:

We might want to name the functions laspoints/getpoints or something like that if we want to export them.

visr · 2018-06-24T10:35:44Z

src/header.jl

    variable_length_records::Vector{LasVariableLengthRecord}
    user_defined_bytes::Vector{UInt8}
 end

 function Base.show(io::IO, header::LasHeader)
-    n = Int(header.records_count)
+    n = Int(header.records_count_new)


I guess that here you need to do something similar as above?

n = lv >= v"1.4" ? header.records_count_new : header.records_count

visr · 2018-06-24T10:45:07Z

src/fileio.jl

@@ -128,15 +131,15 @@ end

 function save(s::Stream{format"LAS"}, header::LasHeader, pointdata::AbstractVector{<:LasPoint})
    # checks
-    header_n = header.records_count
+    header_n = header.records_count_new


Same question here

visr · 2018-06-24T10:56:44Z

src/header.jl

@@ -64,7 +64,7 @@ mutable struct LasHeader
    point_return_count_new::Vector{UInt64}  # 15

    # VLRs
-    variable_length_records::Vector{LasVariableLengthRecord}
+    variable_length_records::Dict{UInt16, Union{LasVariableLengthRecord, ExtendedLasVariableLengthRecord}}


Why do you prefer to make the Vector into a Dict?

Would be good to mention here what the key is supposed to be. It is Record ID right?
That means we cannot store two of the same Record IDs, I don't see why that should not be allowed. Also from the LAS 1.4 spec:

The Record ID is dependent upon the User ID.

Different User IDs are likely to use the same Record IDs.

visr · 2018-06-24T10:58:25Z

src/LasIO.jl

@@ -8,7 +8,7 @@ using FileIO
 using FixedPointNumbers
 using ColorTypes
 using GeometryTypes # for conversion
-using Compat
+using StaticArrays


REQUIRE needs to be updated for this

(cherry picked from commit c78d207)

Data taken from https://github.com/laspy/laspy/tree/master/laspytest/data.

visr · 2018-12-06T14:00:16Z

Rebased, this branch now runs on julia 1.0 as well.

c42f · 2018-12-07T04:45:57Z

Cool stuff guys, sorry I can't help with it (don't have the time... I no longer use this stuff at all!)

Random additional thoughts on lazy loading:

las_file = LasIO.open("test.las") # create `LASFile` stream, reading only the minimal amount (header without VLRs?)
# Following possibly return iterators?
# How to get points to associate neatly with waveforms? `mmap` would help a lot there.
p = collect(points(las_file))
v = collect(vlrs(las_file))
h = header(las_file)

data = LasIO.load("test.las") # just load all the data as in the old interface
# Could return a `LASData` instead of a tuple?
p = points(data)
h = header(data)
v = vlrs(data)
w = waveforms(data)

# It's a bit weird, but if `LASData` supports iteration, destructuring can be made to work
# (not suitable for production code which will have to deal with optional waveforms)
header,points = LasIO.load("test.las")

The function names there are a bit generic, not to be taken too seriously ;-)

evetion changed the title ~~Las 1.3 and 1.4 read support. [WIP]~~ [WIP] Las 1.3 and 1.4 read support. Apr 1, 2018

visr reviewed Jun 24, 2018

View reviewed changes

evetion and others added 13 commits December 5, 2018 17:57

Las 1.3 and 1.4 read support. [WIP]

de09ae9

Add support for reading EVLRs.

5ae83fc

First methods for custom point types.

1729e2b

Fixed small typo.

933edd1

First waveform parsing.

66193aa

FixedString{N} type for IO of LAS header fields

9aaa791

(cherry picked from commit c78d207)

Fixed tests.

e516502

VLRs stored in Dict in header. Cleaned up code and tests.

62872c1

Added extra bytes parsing.

fb4a2bd

Fixed syntax errors.

7b22e0f

Added las 1.3 and 1.4 examples for testing.

d9b7a16

Data taken from https://github.com/laspy/laspy/tree/master/laspytest/data.

update branch to julia 1.0

7ad58c5

update README, stop CI on julia 0.7

9691199

visr force-pushed the wavef branch from 2e87503 to 9691199 Compare December 6, 2018 13:58

visr mentioned this pull request Jan 15, 2019

[WIP] First DataStreams version #13

Open

5 tasks

Crghilardi mentioned this pull request Apr 23, 2020

API for returning scaled and offset point cloud #34

Open

evetion mentioned this pull request Nov 23, 2020

Las1.4 support with all point formats #37

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Las 1.3 and 1.4 read support. #16

[WIP] Las 1.3 and 1.4 read support. #16

Uh oh!

evetion commented Apr 1, 2018 •

edited

Loading

Uh oh!

evetion commented Apr 1, 2018 •

edited

Loading

Uh oh!

evetion commented Apr 1, 2018

Uh oh!

visr commented Apr 1, 2018

Uh oh!

evetion commented Apr 17, 2018

Uh oh!

codecov-io commented Apr 22, 2018 •

edited

Loading

Uh oh!

evetion commented Apr 24, 2018

Uh oh!

visr left a comment

Uh oh!

visr Jun 24, 2018

Uh oh!

visr Jun 24, 2018

Uh oh!

visr Jun 24, 2018

Uh oh!

visr Jun 24, 2018

Uh oh!

visr commented Dec 6, 2018

Uh oh!

c42f commented Dec 7, 2018

Uh oh!

Uh oh!

[WIP] Las 1.3 and 1.4 read support. #16

Are you sure you want to change the base?

[WIP] Las 1.3 and 1.4 read support. #16

Uh oh!

Conversation

evetion commented Apr 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evetion commented Apr 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evetion commented Apr 1, 2018

Uh oh!

visr commented Apr 1, 2018

Uh oh!

evetion commented Apr 17, 2018

Uh oh!

codecov-io commented Apr 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

evetion commented Apr 24, 2018

Uh oh!

visr left a comment

Choose a reason for hiding this comment

Uh oh!

visr Jun 24, 2018

Choose a reason for hiding this comment

Uh oh!

visr Jun 24, 2018

Choose a reason for hiding this comment

Uh oh!

visr Jun 24, 2018

Choose a reason for hiding this comment

Uh oh!

visr Jun 24, 2018

Choose a reason for hiding this comment

Uh oh!

visr commented Dec 6, 2018

Uh oh!

c42f commented Dec 7, 2018

Uh oh!

Uh oh!

evetion commented Apr 1, 2018 •

edited

Loading

evetion commented Apr 1, 2018 •

edited

Loading

codecov-io commented Apr 22, 2018 •

edited

Loading