Add default "unknown" Instrument once information is not present

[jomey]

Some observations from layers (i.e. temperature or density) do not come with instrument information. Instead of leaving the instrument_id column empty, we are adding a default "unknown" instrument to keep the DB record integrity. The foreign key in this table is also enforced via non-null constraint.

This was the result of a discussion with @aaarendt
FYI - @micahjohnson150 @micah-prime

[micahjohnson150]

Are we sure there isnt? Depending which pits were talking there is good info in the user guides which I would be for using. Even if it was just a kwarg we passed in.

Ref:
https://nsidc.org/sites/default/files/snex20_gm_sp-v001-userguide_1.pdf

[jomey]

Great catch and thanks for sharing this reference.

I was going by the current upload logic. For example here

[micahjohnson150]

Yeah that was definitely from when I didnt care about instruments at all. If we maintain that, there will be lots of unknown instruments. I cant recall, did we decide what do to do for manual obs like stratigraphy (instrument wise)?

We could restructure it some, glob and batch upload on predefined ones (density, temp) and then let the rest roll.

[jomey]

I think the stratigraphy does not need an instrument, since it will be implicitly logged with the associated information (hardness, grain size, grain type, etc).

Happy to have a look at adding the PDF device info to the upload script.

Still voting to have a fallback logic like this (pseudo code):

# Give priority to passed information from kwargs
if self._instrument:
    instrument_attributes = dict(
        name=self._instrument,
        model=self._instrument_model,
    )
# Look at headers
elif metadata.instrument:
    instrument_attributes = dict(
        name=metadata.instrument,
        model=metadata.instruemnt_model,
    )
# Got nothing
else:
    instrument_attributes = dict(name=self.UNKNOWN_INSTRUMENT)
    self.logger.warn("No instrument found with profile")

return self._check_or_add_object(
    session, Instrument, instrument_attributes
)

This block would be executed in case there is no info with an individual row

[micahjohnson150]

Makes sense to me, but doesn't it defeat the purpose of the have the non null constraint? Don't get me wrong I am sure we will probably find places where its truly unknown.

[jomey]

Makes sense to me, but doesn't it defeat the purpose of the have the non null constraint? Don't get me wrong I am sure we will probably find places where its truly unknown.

You do have a point there.
So you would rather see us passing the "unknown" instrument in via the uploader script?
I can see an argument for that.

[micahjohnson150]

Yeah...I think so? At least that way we could have a verbose way to demonstrate we didn't know it at the time we wrote it and could easily be fixed at the uploader level when someone does figure it out. Probably the most sane and explicit way.

[jomey]

One case I am running across now that gets tricky is the stratigraphy. There we have a mix of instruments (crystal card, scope, glove) and all the data is in one CSV. For now, I will use the "manual" instrument for that and will worry about this special case in a separate PR.

[jomey]

Another tricky case:
LWC sensor serial number is in the site_details.csv, but not in the LWC.csv

This is all Grand Mesa data btw.

[micahjohnson150]

Separate PR makes tons of sense. Perhaps we just pass a dictionary of types and instruments? IDK.

Yeah LWC...and I am sure others will cause consternation. The real solution is to have an uploader that simply looks at all the LWC files and puts together the instrument combinations. BUT that sounds like a future us problem lol. If its not being prescribed in a user guide or a the current file's header, lets triage that stuff with unknown for now. We wont be able to address all the nuances of each dataset where instrument descriptions live now. And there will be more I am sure.

Conversely, we could ignore serial numbers, which may simplify a lot.