missing_constant treated differently to valid_minimim/valid_maximum

[rjwilson-LASP]

The Validator tool gives me errors about the missing_constant value here, for a number in CSV files to 3 decimal places.
e.g. I'd assume the xml code should be this:

        <Field_Delimited>
          <name>name here</name>
          <field_number>38</field_number>
          <data_type>ASCII_Real</data_type>
          <field_format>%10.3f</field_format>
          <unit>eV</unit>
          <description>
            Species 3 Temperature Uncertainty
          </description>
          <Special_Constants>
            <missing_constant>   -1.000</missing_constant>
            <valid_maximum>   10000.000</valid_maximum>
            <valid_minimum>       0.001</valid_minimum>
          </Special_Constants>
        </Field_Delimited>

The validator gives warnings every time the CSV has a -1.000 in that column.
Yet if I alter one line above to be:

            <missing_constant>   -1</missing_constant>

The validator is then happy, despite the CSV being unaltered with -1.000's in it.

I've had similar issues with column formats of %10.3f, %9.4f and %8.6f, all with missing_constant values of -1 (to however many decimal places).
But weirdly, one column had a format of %9.3f (with a missing constant of 99999.999) and this didn't complain. (perhaps because not negative?)

That seems unexpected, and I'd prefer to keep all values here to the same precision.
Is this a bug in the code?

[rjwilson-LASP]

I get a similar sort of error with:

          <data_type>ASCII_Real</data_type>
          <field_format>%14.7E</field_format>
          <Special_Constants>
            <missing_constant>-9.9999999E+04</missing_constant>
            <valid_maximum>    1.0000000E-08</valid_maximum>
            <valid_minimum>    1.0000000E-17</valid_minimum>
          </Special_Constants>

Which the validator warning of: "Field has a value '-99999.999' that is less than the defined minimum value '1.0000000E-17'. "
I thought missing_constant could be outside of the valid_min/max?

[jordanpadams]

@rjwilson-LASP what version of validate are you running?

also, just curious, how did you create this ticket? It looks like it is missing an issue template, so curious how was created without one.

[rjwilson-LASP]

Hi, am using 3.7.1 which gave both the above issues, but had 3.6.3 previously and was getting the first issue. (The second I only made and tried an XML file after I had 3.7.1).
As for how I reported this, I was on the GitHub website, clicked issue and started one. Not sure exactly what the button was, and I am generally unfamiliar with GitHub, so extremely possible I did something the wrong way.

[al-niessner]

@jordanpadams can we trim the white space of a special constant? My reading is no because it is supposed to be what the user types is literal truth - remember the special constant is not the same type as the cell nor is required to be. Example: fixed width column of 3 characters. missing constant is a. Trimming would give 'a'. However a valid field would be a that also trims to 'a'. Silly, yes. Still, it demonstrates why we should not trim.

@rjwilson-LASP

There are fixable and non fixable items in your statements. First, comparing special constants with values in a table or array are tricky because their respective types do not need to be the same according to the PDS documentation.

The special constant -1 works but -1.000 does not. It is a floating point conversion problem. When validate asks Java to convert -1.000 from the special constants it gets a different bit pattern then when converting from the CSV. I have no idea why other than -1.000 does not have an exact floating point representation causing a == b to fail. When given -1, validate converts it to an integer allowing a == b in this case.

Validate compares the special constants and values as strings before doing int and float conversions. By removing the white space in <missing_constant> -1.000</missing_constant> it will do a better job. It may still fail if your CSV does not represent it as -1.000 (say -1) as you are back to float conversions. I would test this but I do not see a data set to work with on this ticket.

The same is happening with -9.9999999E+04. The imprecise float representation of that number makes it one bit away from -9999.999 and thus fails. If you changed the special constant to <missing_constant>-9999.999</missing_constant> - assuming that is how it is in your CSV - then it should work.

I understand that the white space in your XML is an attempt to make the XML more readable and removing it is not ideal. I also understand that your choice of how values are represented may have meaning as well -(-9.999999e+04 vs -9999.999). However, the problems you are experiencing are bound by the limitations of floating point quantization and to avoid those limitations changing the representations may be the only solution.