Provide guidance on System Identifier for tile-based data

[esilvia]

I received an email from the USGS QC team indicating that the System Identifier (SystemID) field in the LAS header quite often is something other than the sensor used to collect the data. The spec is deliberately vague on its actual implementation, and in my opinion that's a good thing.

I believe it's clear how to implement it for a file containing a single flightline, but it's unclear how this field can be best utilized if data from multiple sensors are in the LAS file. For example, 32 characters is barely enough for something simple like "Leica ALS80 & Riegl 1560i" but if serial numbers are desired it gets too long.

Does anyone have experience with a good multi-sensor implementation they're happy with that we can recommend? I'm not sure that this merits a change to the specification, but we can perhaps provide guidance to USGS for their contractors.
@jstoker74

[rapidlasso]

It seems that these strings often contain something that has nothing to do with the actual hardware that was scanning. Here are a few random ones from my debug sample folder:

  version major.minor:        1.2
  system identifier:          ''
  generating software:        'GeoCue LAS Updater'
  file creation day/year:     305/2016

  version major.minor:        1.2
  system identifier:          'LAStools (c) by rapidlasso GmbH'
  generating software:        'las2las (version 170106)'
  file creation day/year:     155/2013

  version major.minor:        1.2
  system identifier:          ''
  generating software:        'TerraScan'
  file creation day/year:     125/2017

  version major.minor:        1.4
  system identifier:          'OTHER'
  generating software:        'LP360 from QCoherent Software  '
  file creation day/year:     315/2016

  version major.minor:        1.2
  system identifier:          ''
  generating software:        'LASzip DLL 2.4 r0 (150731)'
  file creation day/year:     318/2016

  version major.minor:        1.2
  system identifier:          'Riegl Q680i'
  generating software:        'Terrascan'
  file creation day/year:     29/2014

[jstoker74]

@esilvia I'm not sure if this is possible, but perhaps an agreed upon standardized shorthand/coded list maintained on the wiki page would allow you to incorporate more than one sensor in the space provided? Also would allow for a standardized list of how sensors should be populated in the field? e.g. Leica ALS80 vs ALS80 may appear to be two different kinds of sensors in a meta analysis query.

[hobu]

What about this combo?

• A registry wiki page that allows vendors to update their system_id with a key and whatever information they want to provide

• An automated process that grabs that wiki page, snapshots it to a point in time, and includes that in an "errata" section at the end of the specification (with appropriate caveats).

[esilvia]

@hobu +1

I like the sound of this. It'd be even more useful if we could link the sensor's tech specs pages from there!

@csevcik01 Is this something that Riegl would be willing to contribute to? We can start with the airborne sensors.

[milenajaniec]

This is to revive the discussion regarding the system identifier and brainstorm how the information could be presented.
I had the following suggestions:

1. To make it in such a way, that an average user can easily look up which systems were used during the collection. This look-up table could be kept in an online repository. @jdnimetz commented that “We would need to define who manages this table. It would probably fall to the ASPRS LD.”
2. To easily identify platform type. Presently, some systems can be fitted into different platforms. We could signify platform type (Spaceborne /Airborne - Manned /Airborne - Unmanned /Terrestrial – Mobile/Terrestrial – Stationary), use a simple code to distinguish between them). This would give the user an idea how the system was used for a particular project. Currently, for the majority of the systems, this is obvious. However, we may be moving in the direction of greater interchangeability. Perhaps we could take an advantage of this field to give the user a bit more information about data collection?
3. To be able to fit more than four systems. This is to accommodate folks who worry that they cannot populate this field with meaningful information due to the lack of space.

Overall, a user could read this field as a sentence that states "For this particular project we used manned platform (for another project we could have utilized a different platform), systems’ names were abc and cba"

In response, @esilvia proposed a simple and elegant solution, by simply using abbreviations which could accommodate multiple manufacturers and models. Here is Evon's example:

System_ID_codes_brainstorming_November_ES_JDN_MJ.docx

4. If we were to use just the abbreviation for the system and differentiate between platforms, we should be able to fit up to six systems.

I think that having this communicated in a form of a look up table would make it accessible for an average user to find rudimentary information about how the data were collected.

[jdnimetz]

@milenajaniec
Per our discussions, I am in favor of a look up table for defining system identifier. However, I suggest ASPRS PDAD may be more appropriate entity to manage this table.

[milenajaniec]

Thank you for clarifying this @jdnimetz .

[esilvia]

We've been able to refine this suggestion somewhat.

System_ID_codes_brainstorming_November_ES_JDN_MJ_ES.docx

The thought would be to combine a single-digit letter for the platform followed by a 4-digit code that corresponds to the sensor itself. For example, a Riegl VQ-880-G mounted in a fixed-wing aircraft would be designated by the code "AR881".

Up to five space-delimited sensors can be added to a SystemID in this way. For example, if a Leica ALS80 mounted in a plane and a Velodyne VLP-16 mounted in a helicopter contributed points to a tile, the SystemID for that tile would be "RVP01 ALA80".

---

Is this getting too cumbersome and arcane? How is this strategy better than a VLR with a null-terminated list of system names and serial numbers? We could designate a special SystemID that indicates a user should look for that VLR.

[milenajaniec]

@esilvia Thank you for posting the look up table. As you know, personally, I think this would work.

[milenajaniec]

Please review the latest version:
System_ID_codes_brainstorming_November_ES_JDN_MJ_ES_v2.docx

[esilvia]

@milenajaniec I believe that QSI has delivered a couple datasets using these codes now. Is it working? Does anyone have any remaining input before we start rolling this out more officially?

[milenajaniec]

@esilvia I asked the person who oversees the evaluation of one of the projects, for feedback. I have received the following response:

"1. Hypothetically, how would one platform with two systems be displayed?
2. Please be clear that the first character location refers just to the platform, and the rest refers to the system.
3. Please have space, underscore, or some other type of an interruption between different system identifiers, to make it human readable."

However, the table itself met with approval. I received a comment that it was helpful and information was easy to locate. It seems that it provided much needed meaning to the system identifier bytes.

Still, it seems, that the way that the scheme works for different systems should be explained in detail. I believe, that we have discussed how to arrange multiple system identifier codes, for instance, using space between them (ALA80 ALA70) and including the platform for each sensor to avoid ambiguity. Nonetheless, it was conveyed to me that we should make sure to describe all of this in detail within wiki.

[esilvia]

Added three systems from a recent multibeam sonar survey, and a generic ID for MultiBeam Sonar in general. I couldn't think of a sensible way to do it intelligibly so I went with more generic codes of the format "MBxx".
System_ID_codes_brainstorming_November_ES_JDN_MJ_ES_v3.docx

[esilvia]

It was pointed out to me that the Ross Labs system is a Sonar Sweep system, not a multibeam system. As such, its code has been changed from MB03 to SR01.
System_ID_codes_brainstorming_v4.docx

[nkules]

To throw a wrench into this whole discussion of system ID codes. It was discussed that the entire purpose for this field should be for casual user lookup.

In that spirit, how many casual users can actually explain the meaningful differences between lets say an ALS80 and a Riegl 1560? In my experiences most of these end users have more of a marketing based mindset than anything useful they'd gain from actually knowing the sensor model.

With that, I'd suggest that specifying sensor technology is much more useful to the average user. Defining "Airborne linear mode lidar" vs "Terrestrial phase shift lidar" would tell much more of a "need to know" story about the LAS file. In addition this is far more future proof and a "specification" based approach with having some generalization.

I just would want to avoid it turning into a mess of codes that no users will actually look up. These more specific details are always available in metadata.

[milenajaniec]

@nkules Thank you for taking time to add to the discussion. Honestly, I do not know how many users will know the difference between specific systems, such as ALS80 and a Riegl 156. I can only speak for folks who are seeking a more helpful information to interpret las files. We are looking for the system identifier field to be populated in such a way that a person could look it up and then, possibly understand why the point cloud looks a certain way (denser towards the edge of the swath, etc.), or why the Scan Direction Flag is populated the way it is. In other words, populating system identifier field in more meaningful way may help to explain why some of the other fields in a point cloud are filled the way they are. A lot our people are scientists, and they would like to gain as much information as possible. Ideally, yes, there should be metadata available with those files. However, in real life it is sometimes unavailable.
Also, we were thinking how to accommodate files containing point cloud derived from multiple sensors and platforms and fit all that information within the character limit.
I could be wrong, but if the table existed, and was available online, as a user I would just type the “code” and “system identifier” into a search engine and hoped to be directed towards the page with the table. For example, in the same way I can type in a code and method number for the USGS turbity parameters and the USGS look up table will pop up.
However, yes, we agree that knowing the platform can be particularly meaningful. Therefore, the first part of the code is a letter corresponding to the platform.
Please let us know.

[jtellini]

I think that having a way to include as much information as possible about the sensor into this field, such as using the system ID codes table, would be more inclusive and beneficial to users in general.

[nkules]

I definitely see the advantage of including the maximum amount of information. I like that this is a very technical solution. My issue with this approach would primarily be the need for a user to look up additional information to decode, as well as not being a future proof solution. This solution requires A) that someone maintains the look up tables and B) the ability to encompass all sensors that might use the LAS format. This puts the effort on those supporting the format instead of the user/creator of the files.

I might be preaching to the choir here but my mindset when approaching the LAS format would be to build in as much longevity as possible. If you look at longstanding formats such as tiff, it has survived without a major release in 27 years (not counting extensions). The LAS format is obviously more complex and inherently more dynamic so it would be crazy to expect that sort of longevity. However I feel that keeping some level of abstraction will at least lend to nurture this.

In the case of additional sensor information (this is probably going beyond the scope of the specification discussion) has anyone considered embedding this information into VLRs (or ELVRs)? This would allow users to burn in metadata information such as sensor type(s) or even all of the fields that are encompassed in the USGS tags. Reading the VLRs are fairly quick and easy and some packages out there (e.g. Lastools Lasinfo) already read them as part of the header scan.

That opens the door for the excessive use of VLRs, but just an alternative thought.

[milenajaniec]

@nkules Definitely the lookup table would require maintenance on wiki. Either folks would submit their preferred code for a new system (or one that was accidentally omitted) that we could add to the table, or we could assign one.
I am not sure if I understood your question regarding longevity. We are not asking to change the System Identifier (SystemID) field in the LAS header. We are only asking for guidance for people who would like to populate the field in a consistent manner and maximize its use.
Certainly, Lasinfo provides a lot of help in terms of reading the VLRs. And, of course, if someone wishes to include additional /specific information, by including it in VLRs I (subjectively) have no objections. But, yes, there should be probably some consideration given to how much use of VLRs is too much and what would be just right 😊
Still, this is about a 32 character field that already exists and is waiting to be populated with relevant, consistently organized information. Potentially, this document could be a guidance for folks on how to do that.

[esilvia]

@nkules Thanks for the great discussion here! To address your concerns about longevity, nothing in this entire thread actually translates into a material change of the specification. I imagine that most users of the LAS file will ignore (and has ignored, if we're being honest) this entire discussion and continue to use the SystemID in whatever manner they wish. That methodology is 100% spec compliant and will receive no criticism from me.

If, however, an entity such as USGS wishes to encode more meaningful information into the SystemID field, supplying this table and convention in the wiki would give them the option to point to the table and say "do that" and receive data that they already know how to interpret. That's the purpose of the wiki pages here.

If in 5 years the LWG has disappeared and the table is no longer maintained, the viability of LAS isn't threatened because usage of the SystemID would revert back to its previous usage.

It's interesting that you bring up the TIFF spec because it actually has a very similar parallel. The GeoTIFF specification provides a mechanism for encoding CRS information that's remained largely unchanged for many years. It was so stable, in fact, that it remained the method for CRS encoding in LAS until LAS 1.4. However, as new coordinate systems (e.g., UTM10 NAD83(2011)) entered the world, a governing body had to introduce new codes that could be fed into the unchanging structure provided by GeoTIFF. That burden largely fell on the EPSG strictly by convention, although technically anyone could have done it.

In the same way, the LWG can maintain a set of codes to populate into the SystemID until such a time as it no longer serves the community. That set of codes seemed easier to maintain than a VLR since it didn't require changing anyone's LAS readers.

[esilvia]

A while back I received a contribution of several camera systems, which I've added. I also added generic codes for all of the modalities I could think of. I also added conditional formatting to highlight non-unique cells.
System_ID_codes_v5_20200921.xlsx
I'll bring this up on the call today. If approved, I'll convert it into a wiki page.

[esilvia]

@milenajaniec
I got a start on the wiki today and should be able to finish it later this week. Can you take a look and let me know if there's anything big that should be changed?

(edit) link: https://github.com/ASPRSorg/LAS/wiki/Standard-System-Identifiers

This is the xlsx it's based on.
System_ID_codes_v6_20201215.xlsx

[milenajaniec]

Thank you @esilvia ! I think this will work very well for us. I think we should also add L3Harris' GML system (and we can add more later on, but this is what I could think of for now). @jdnimetz please let us know.

[esilvia]

I'd be happy to add more sensors. @kdamkjer can you possibly contribute the formal names for your GML systems? In the absence of your input I'll call them Harris GML1 (HG01) and Harris GML2 (HG02). Are there others? Would you like them called something different?

[esilvia]

I added the two Harris sensors and one from Fugro.

[milenajaniec]

Thank you @esilvia - I appreciate your work on this!

[esilvia]

I finished up the wiki and removed DRAFT markings. Today I added example LAS files and SystemID values for multiple different combinations.

The "final" draft of the wiki is here: https://github.com/ASPRSorg/LAS/wiki/Standard-System-Identifiers

I noticed that I missed codes for the Optech Lynx mobile systems. If anyone knows someone at Teledyne Optech I'd appreciate a connection to them so that we can ensure that all of their systems are represented... I'm not personally familiar enough with their systems to take a stab.

Hexagon/Leica is also unrepresented on the LWG now. Does anyone know someone?

@csevcik01 Can you or one of your colleagues review the list of Riegl systems and contribute any that might be missing?

@mattbcolorado Does Merrick have any lidar systems that should be added?

@jdnimetz Do you have data from any recent systems that I'm missing?

If I don't hear from anyone by EOY I'll close this thread.

[manleypv]

@esilvia I can help you with the Optech connection

[rapidlasso]

Is anyone already reaching out to manufacturers of cheaper LiDAR systems, like Livox, Ouster, Quanergy, ... that start populating the lower-end UAV market?

[manfred-brands]

@esilvia Small grammar error in point 2: (remove word can)

It is unclear can how to encode

[kdamkjer]

Evon, There is only one commercial model of L3Harris sensor. We can call it “L3Harris Geiger-mode LiDAR”. Also, please update Harris Corporation to L3Harris Technologies. I think the HG01 designator is fine. For the “Pure Generic” sensors, we may want to differentiate between Geiger-mode APD and PMT lidar sensors since they have different product characteristics (instead of having a single “photon counting lidar” category). Also, all lidar are TOF, so perhaps “linear-mode” or “standard lidar” would be a better term? Especially to avoid confusion with TOF cameras. Please let me know if I can provide any other assistance. Kristian L. Damkjer, PhD, CMS-LiDAR Lead, Software Engineering, Geospatial Software Space & Airborne Systems / L3HARRIS TECHNOLOGIES t +1 321 984 5773 / m +1 321 890 4198

[mattbcolorado]

Merrick uses COTS LiDAR systems, nothing proprietary. Matt Bethel, GISP | Director of Operations and Technology | Merrick & Company T: +1 303-353-3662 | C: +1 720-979-4451 | S: matt.bethel1 | www.merrick.com<http://www.merrick.com/> Engineering | Architecture | Design-Build | Surveying | Planning | Geospatial Solutions

[esilvia]

There is only one commercial model of L3Harris sensor. We can call it “L3Harris Geiger-mode LiDAR”. Also, please update Harris Corporation to L3Harris Technologies. I think the HG01 designator is fine.

Fixed corporation name and sensor name in the tables and attachments. Also fixed in the 5-sensor LAS example, and removed the nonexistent GML2 sensor.

For the “Pure Generic” sensors, we may want to differentiate between Geiger-mode APD and PMT lidar sensors since they have different product characteristics (instead of having a single “photon counting lidar” category). Also, all lidar are TOF, so perhaps “linear-mode” or “standard lidar” would be a better term? Especially to avoid confusion with TOF cameras.

You're right that TOF is misleading, and I think it's trying to be too granular. I'm just trying to capture linear scanning systems vs. array-based solid-state systems, as their behaviors are wildly different. Is there a sensible way to do that?

[esilvia]

@manleypv Feel free to send them a link to www.lasformat.org or shoot me an email at [email protected].

@manfred-brands Thanks! Fixed.

@rapidlasso I have no contacts there. Feel free to send them my way if you do. My hope is that people who begin to use those and the massive volume of camera systems out there will reach out when they attempt to create LAS. Most won't generate LAS files at all, so it'd be wasted effort trying to support them preemptively.