Large unsigned integers (>long.MaxValue) are parsed as floats.

[xlsupport]

My json contains ulong values, but the parser incorrectly converts values that exceed the maxvalue of a long to floats.

I'm assuming it happens here [https://github.com/molsonkiko/JsonToolsNppPlugin/blob/d45a7e5ac491af6dbf4f2bc2f381797cff5984b1/JsonToolsNppPlugin/JSONTools/JsonParser.cs#L1257]

[ {
    "State": 8598537377453965584,
    "Moved": 17592454483968
  },
  {
    "State": 9601714194450817024,
    "Moved": 76842668642009376
  }
]

becomes
[
    {
        "State": 8598537377453965584,
        "Moved": 17592454483968
    },
    {
        "State": 9.60171419445082E+18,
        "Moved": 76842668642009376
    }
]

[molsonkiko]

@xlsupport

Sorry, but I am not going to special-case integers between long.MaxValue and ulong.MaxValue, because that would require way too many updates to the code base, and would probably introduce a variety of subtle bugs.

If you really want to manipulate JSON in Notepad++ without converting integers into floats, you might consider PythonScript, since Python uses unbounded-size integers.

[xlsupport]

Your choice, but Json in C# parses it correctly, so why shouldn't your tool?

[molsonkiko]

@xlsupport

Your choice, but Json in C# parses it correctly, so why shouldn't your tool?

In an ideal world, my tool wouldn't even screw around with longs or ulongs, but instead use Python-style unlimited-size integers.

In the real world, I'm working full-time, I'm not super motivated to make very involved overhauls to this project, and nobody is offering to do it for me.

[molsonkiko]

Good news! I found System.Numerics.BigInteger, which appears to be the arbitrarily large int I'm looking for. If performance seems acceptable, I'll probably refactor and incorporate that in my next release.

[molsonkiko]

@xlsupport
Well, after spending way too long hunting down stupid little bugs, I have re-implemented JsonTools to use BigIntegers instead of longs in this branch.

Some highlights from the benchmarks:

• integers may take 6 times as long to parse
• numeric operations in RemesPath seem to take about twice as long
• In this nested JSON file where about 40% of values are integers, parsing was about 60% slower

Obviously I'm the final arbiter, but given the aforementioned losses in performance, do you think it's worth merging these changes into main?

[xlsupport]

Anything that significantly slows down existing code needs careful consideration.

BigInteger is intended for Numerics/Math. Maybe using it for JsonTools is like using a chainsaw to slice tomatoes.
Can't you tell it's a tomato before going to the shed to fetch the saw?

I'm just a casual user of JsonTools who reported a bug when parsing Uint64.
In case you hadn't noticed.. it's summer ;-) Leave it!

[molsonkiko]

Alright, well I just implemented a benchmark that tests how long it actually takes to parse arrays of integers.

TLDR: Although benchmarks for parsing small arrays of integers suggest that BigInteger parsing performance is up to 10 times worse than long parsing performance, benchmarks using extremely large (200-300MB) files suggest that for large files containing only integers, BigInteger parsing is only about 2-5 times slower than long parsing.

Here are the results for that benchmark on the current main (using 64-bit integers):

In 256 trials, it took 383 +/- 106 μs to parse an array of 1028 integers with absolute value greater than 2**32 - 1.
In 256 trials, it took 305 +/- 102 μs to parse an array of 1028 integers with absolute value less than 2**32 - 1.

And here are the results for that benchmark on the BigInteger branch:

In 256 trials, it took 2653 +/- 534 μs to parse an array of 1028 integers with absolute value greater than 2**32 - 1.
In 256 trials, it took 1228 +/- 277 μs to parse an array of 1028 integers with absolute value less than 2**32 - 1.

However, since arguably we only care about this performance issue for very large files, I did yet another comparison, with the following results:

BigInteger implementation is at https://github.com/molsonkiko/JsonToolsNppPlugin/tree/cc84249470f439cdc32ac47c4b3dcd974dbe74c0
long implementation is at https://github.com/molsonkiko/JsonToolsNppPlugin/tree/v8.4

For each case, four trials were done. The times listed below are in seconds.

Pretty-printed array (approx. 226MB) of 16000 arrays of 1028 integers (each between -2**31 and 2**31 - 1):
BigInteger implementation: 19,16.5,15.8,16.2
long implementation: 7.0,7.3,7.0,7.5

Pretty-printed array (approx. 332MB) of 11000 arrays of 1028 long integers (each between -2**63 and 2**63 - 1):
BigInteger implementation: 48,47,48,40
long implementation: 13.5,11.3,7.8,8.1

=======
To create the arrays, use this below in PythonScript 3.10.11 (replace the 1<<63 with 1<<31 for the small integer case)
import random, json; l = [[random.randint(-(1<<63), (1<<63)) for ii in range(1028)] for jj in range(11000)]; editor.setText(json.dumps(l, indent=4))

[molsonkiko]

After some more thought, I've concluded that the best solution to this problem is to maintain the big_integer_for_Dtype_INT branch separately from main and just include a reference to it in the documentation, so that anyone who works with JSON containing very large integers can use that if they need it. This way nobody who doesn't need the feature will have to suffer major performance losses.

At some point in the future I may revisit the feasibility of using long's and BigInteger's both in the same JsonTools version, but I suspect that doing so would make the code far too difficult to maintain.

The big_integer_for_Dtype_INT branch would receive bugfix updates to address any major issues reported by users, but would not receive any new features. This is probably not a major loss, since I regard this plugin as being in the maintenance stage of development anyway.

[molsonkiko]

@xlsupport

OK, I believe that this will be my last update and I will at last leave you alone.

Just wanted to mention that I have now released v8.4.0.4 as the first official release that supports big integers. It is referenced here in the docs.