Should we make a clear value-level distinction between floating and integral numbers?

[leftaroundabout]

This basic subject has been variously discussed before –

#227
#181

The thing is that Aeson can't make a distinction between integer values and floating-point values (scientific values) which merely happen to be equal to an integer.

Prelude Data.Aeson> encode 4
"4"
Prelude Data.Aeson> encode 4.0
"4"
Prelude Data.Aeson> encode 4e+30
"4000000000000000000000000000000"
Prelude Data.Aeson> encode 4.53986e+30
"4539860000000000000000000000000"

The former two may be sensible enough. The latter are kinda correct too, at least in a dynamically-typed understanding of numbers; but they're arguably not desirable. It's most obvious when interpreting JSON as a human-readable format, as the aeson-pretty library does. There, this behaviour is under discussion right now. In science, numbers like 4.53986e+30 are very much not something unusual, and nobody would consider it acceptable to print them in expanded, exact integer form if the purpose is human readability.

Now, it's understood that Aeson itself does not really focus on human readability, but even so, such insignificant-digit-monsters are a bit of a pain. The space usage and parsing overhead can't be completely irrelevant.

Dynamic languages like Python actually make a distinction between integers and integral-floats in JSON:

Python 3.4.3 (default, Nov 17 2016, 01:08:31) 
Type "copyright", "credits" or "license" for more information.

IPython 4.0.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import json

In [5]: json.dumps(4)
Out[5]: '4'

In [6]: json.dumps(4.0)
Out[6]: '4.0'

In [7]: json.dumps(4e+32)
Out[7]: '4e+32'

Aeson used to have such a distinction too: before aeson-0.7, Number used Data.Attoparsec.Number, which has separate cases. Now granted, this can lead to some problems too

sol/aeson-qq#4

but I think all in all I think we should follow Python here. So would it be viable to try something like

data Value = Object !Object
           | Array !Array
           | String !Text
           | Scientific !Scientific
           | Int !Int
           | Bool !Bool
           | Null
             deriving (Eq, Read, Show, Typeable, Data)

to have a clear grasp of which numbers are really integers and which ones are merely within experimental error in the range of an integer?

[ksromanov]

It is possible to maintain backward compatibility with

data Value = Object !Object
           | Array !Array
           | String !Text
           | Number !Scientific -- represents either integer, or floating point
           | IntegerNumber !Int -- represents only "small" integers
           | Bool !Bool
           | Null
             deriving (Eq, Read, Show, Typeable, Data)

If we do not touch decoding, the only compatibility issue will be a warning in pattern-matching. This is dangerous only if the library, which uses aeson, is compiled with -Werror in release (this is an apparent error in build scripts).

[leftaroundabout]

Actually, I deliberately didn't keep the name Number because for this proposal to work out properly, we must break backwards compatibility: right now everybody wraps their Int values in Number, but if they keep doing that and we format the Scientific values in decimal-fraction-exponent form this will then break applications whick expect proper int values further down the road. Can't have that. With two entirely new constructors, people must be explicit and actually choose the right constructor for their application.

Most applications probably don't handle the Value ADT manually anyway, but let some Generic-derived FromJSON instance do that for them.

[leftaroundabout]

An entirely different point to think about: having a dedicated constructor for Int values might well improve performance, certainly memory usage, in lots of applications, because very often Int is all you need really.

[ksromanov]

The issue ".0 is added to integers or not added to doubles" is much less important than backward compatibility. People rely on aeson, and if it breaks, they will definitely not be happy.

You should take into account that aeson not only encodes, but also decodes JSON. And JSON format is designed to treat all numbers as Scientific (I personally think that it is a mistake, but we can't get 30 years back a change JavaScript and show him ML). So, encoding integers without trailing ".0" and doubles with is a hack.

So, why not to have a recommendation and let authors of the libraries choose a proper format?

[leftaroundabout]

On the decoding front it's even clearer, isn't it? We definitely want to decode small numbers without fractional part as IntegerNumber, but to handle that, people must follow the warning and actually deal with that constructor. (If they even look for the warning; I personally tend to not use -Wall and deliberately employ incomplete case matches as well-defined points of failure for “my-brain-exploded”–this-can-never-happen cases.)

As long as we permit to just leave out the new constructor, there will be some instances of runtime crashes. It is such runtime errors that would be really bad for the people who rely on aeson. OTOH, if they simply can't compile with the new version, they'll know they must stick to an older one or else change their code to incorporate the new cases.

If that's still too much compatibility broken, there's also the (perhaps controversial) option of reinstating Number as a pattern synonym that will handle both floating and integral numbers:

{-# LANGUAGE PatternSynonyms, ViewPatterns #-}
pattern Number :: Scientific -> Value
pattern Number n <- (matchNumber -> Just n)
 where Number n
         | fromIntegral n' == n  = Int n'
         | otherwise             = Scientific n
        where n' = round n

matchNumber :: Value -> Maybe Scientific
matchNumber (Scientific n) = Just n
matchNumber (Int n) = Just $ fromIntegral n
matchNumber _ = Nothing

[phadej]

I have a concern: the JSON spec doesn't differentiate between how numbers are encoded. /Some/ libraries in other languages do differentiate (like Python mentioned above), but I don't think it's a good argument.

Taken to extreme, one might want to differentiate between 1.23 (Centi) and 1.230000 (Micro).

IMHO these discussions would have better ground if the equivalence of JSON documents would be well-defined in the spec. As to my understanding the lowest common denominator is _.isEqual functions in JavaScript, which doesn't differ between Int and Double; so why should aeson. (Also what Eq Value would look like)

We could encode numbers differently (e.g. Centi and Micro to always append digits), but doing something smart with Double will affect encoding performance.

I don't think that parsing of 4539860000000000000000000000000 or 4.53986e+30 differs significantly in performance. aeson should be safe to use on untrusted input. We cannot make some format perform better, sacrificing performance of other one: it would be a security bug.

Prove me wrong with an actual benchmark (e.g. some array of arrays of such numbers, something the scientific data dump might look like).

[ksromanov]

@phadej - of course, parsing takes the same time, but expanded version of 4.53986e+30 is apparently twice longer. But this is not the major problem - suppose, input data of your program contains a Double with garbage (it is bad, but, unfortunately, does happen). It is very likely that its exponent will be around 300 - the expanded version in JSON will span couple lines full of zeroes. I.e. the size will be ten times larger.

Why not to use floating point format, which was specifically designed to safe space and preserve precision? It is just a single line.

@leftaroundabout - we do not know, if JSON number 10 has type integer or double. It might be double, but encoder (not aeson) stripped trailing .0. Or, instead, 3.0 may be an integer with appended .0. Therefore, we can restore type only if we control the source of JSON.

So, by-default decode should parse numbers as Scientific, and only when user of aeson specifically requests this, decode should parse values to both Int and Scientific. If so, Int will be silent, it will not propagate outside aeson if the user did not specifically request it, hence, no run-time crashes.

On the other hand, encode can work with both Int and Scientific, leaving or stripping trailing zeroes.

If so, why should we break compatibility?

[phadej]

The example in first post doesn't specify aeson version, neither the type

WIth aeson-1.1

Prelude Data.Aeson Data.Scientific> encode (1e10 :: Float)
"1.0e10"
Prelude Data.Aeson Data.Scientific> encode (1e10 :: Double)
"1.0e10"
Prelude Data.Aeson Data.Scientific> encode (1e10 :: Scientific)
"10000000000"

IIRC Scientific behavior was made to encode Integers as integers always. 1e10 is number in floating format, and again /some/ libraries might load them as floating point. E.g. Python

>>> type(json.loads("1e10"))
<type 'float'>
>>> type(json.loads("10000000000"))
<type 'int'>

So we cannot encode big Integers in scientific format, because it will break interoperability with Python.

[phadej]

Also

>>> json.dumps(10000000000000000000000000000L)
'10000000000000000000000000000'

[bergmark]

Thanks for the proposal @leftaroundabout,

There seems to be multiple ways this could be implemented and it's not clear to me that a majority wants the same behavior. I primarily work with ints and then I actually don't want there to be any difference between 3.0 and 3.

Everyone is able to make a custom parser and encoder that can handle Scientific in the way they want, so if you have a particular preference I think this is the way to go.

Value represents JSON and I'm wary of extending it to be more powerful than that.

Regarding backwards-compatibility, I do not think there is a way to make such a change non-breaking or easily migrated, I think it would break lots of code in possibly silent ways.

[ksromanov]

@bergmark unfortunately, Scientific looses information about initial type on purpose. So, it is impossible to reliably decide, either to put 3 or 3.0 (for some people this makes it easier to read).

[leftaroundabout]

@phadej taking it to the extreme and covering every number type you could come up with is certainly not feasible – not just with JSON, but with any fixed-specification data format. It can never be possible to distinguish between 0.13 :: Centi and 0.13 :: Double, with just a single JSON-compliant number literal, and it certainly won't be possible to reliable represent it if I come up with a new type that represents exactly, say, all real numbers of the form ±37^{tanh q/12} (where q ∈ ℚ ∩ ]-1,1[).

But who cares about that? If you really need reliable fixed precision, then encode it as integers and make the constant denominator part of the specification. If I really need a crazy, super-precision type that can't be adequately represented as standard number literals, I'll bite the bullet and encode it as a specialised record, which includes all necessary implementation details.

Floating-point numbers are in another book, though. Not only is Double the standard type for representing real numbers in science, it's also the not-quite-but-almost-official recommended type in RFC 7159 for all numbers. If I understood correctly, optimising JavaScript implementations actually store everything in Double, and just cram anything that doesn't fit in a NaN payload of such a number!

Hence one might even argue that it was a mistake in Aeson to ever assume you can safely store high-precision integral values in JSON number literals. Perhaps we should have right from the start have used Number :: Double -> Value, and stored 1 :: Integer as "1"?

(No, I'm not really serious about this, but just saying... support for big integers, compatibility with Python, JavaScript and anything else – can't have, all of that.)

Anyway there is a good case to be made for the need to have good support for actual floating-point numbers in a JSON implementation. And the very essence of floating-point is that precision is magnitude-dependent. It should certainly be exact for small integer values, and I suppose we agree that it should also be precise for bigger integer values, regardless of the previous paragraphs. But being exact in general is completely out of question. It just doesn't make any sense to have precision over 30 orders of magnitude (nobody could possibly measure values at such a precision in the real world), but it's essential to have full precision in each of these orders of magnitude individually. Basically, Double has the property of offering reliably the precision that is sufficient for almost all physical quantities, while at the same time also reliably staying in adequate bounds on memory- and processor usage. In another view, floating-point as a representation of physical quantities should be agnostic of the unit system used, so it shouldn't really matter whether I measure a distance in metres or in light-years. But with an encoder that coaxes any floating-point value into expanded integer notation if it happens to equal an integer within the precision that so happens to be assigned to that order of magnitude, this is totally not given! Memory usage will fluctuate a lot through choice of a different unit system, though that's conceptually a no-op.

Ewww... sorry for the rant. What I'm getting at:

• Value is right now already more powerful than JSON, because that is actually designed with floating-point in mind rather than infinite-precision-integer.
• Yes, maybe it's a hack if we make a pattern match on whether there is a decimal point on the number, to actually choose different Haskell representations. But it's a worthwhile hack, and if anything, this hack will get us closer to existing, real-world applications in other languages, standards conformance and reliability.
• It does make sense to make a distinction between, in particular, inexact-floating-point and exact-integer – whereas other, more specific distinctions between certain fixed-precision fraction types are not sensible.

[phadej]

@leftaroundabout I personally disagree on your second and partially third point.

IMHO the hack isn't worthwhile; if you need to communicate data preserving the types between aeson and not-aeson. you shouldn't use rely on text representation of the numbers (extreme example: don't transfer zip codes as numbers); IIRC either google or amazon encodes id numbers as strings, because double isn't enough to represent 64 bit integer. Not every number is a physical measurement result!

Which leads us to the third point; if you really need to distinguish Integer and Double, don't use JSON.

And I have to repeat: that Double is serialized using scientific notation when it's big or small enough. I don't see any problem here: (aeson-1.1, and 1.2).

Prelude Data.Aeson> encode (4.53986e+30 :: Double)
"4.53986e30"