Handle Large Integers via native BigInt in the TypeScript client

[KtorZ]

• 📍 Use fork / temporary version of json-schema-to-typescript including the bigint fix.
  See Add optional support for TypeScript's bigint bcherny/json-schema-to-typescript#401

• 📍 Use json-bigint parser and serialiser with native bigint.

[...]
export type Epoch = bigint;
export type Hash64 = string;
export type Slot = bigint;
/**
 * A Cardano address (either legacy format or new format).
 */
export type Address = string;
/**
 * An amount, in Lovelace (1e6 Lovelace = 1 Ada)
 */
export type Lovelace = bigint;
/**
 * A number of asset, can be negative went burning assets.
 */
export type AssetQuantity = bigint;
export type DatumHash = string;
export type Null = null;
export type TxWitness = WitnessVk | RedeemWitness;
export type NullableRatio = Ratio | Null;
/**
 * A ratio of two integers, to express exact fractions.
 */
export type Ratio = string;
export type NullableInteger = Integer | Null;
export type Integer = bigint;
[...]

TODO:

• Adjust the client functions to work with bigint.
• Possibly adjust some of the JSON-schema definition to provide a maximum and resort to standard number.

[rhyslbw]

Great solution, hopefully your PR gets merged and we can loose the fork dependency.

Do you want me to adjust the client functions?

[KtorZ]

I don't mind if you do, though we need to adjust the Json shema first, if we want to keep some numbers in there. It'd also be fine to stick to bigint everywhere I suppose?

[rhyslbw]

It'd also be fine to stick to bigint everywhere I suppose?

I think we should implement them selectively https://betterprogramming.pub/the-downsides-of-bigints-in-javascript-6350fd807d

[KtorZ]

@rhyslbw there's currently an annoying thing with the json-bigint library ... It offers two modes:

• Either it parses every numbers as bigint
• Or, it parses numbers as bigint when they are actually bigger than MAX_SAFE_INTEGER

Which means that, even though we may define a particular field or type as bigint, the resulting parsed JSON will be number if the number is small-enough... This is utterly annoying, because now, it becomes very much error prone and causes runtime errors :| ...

See the following example:

import JSONBig from 'json-bigint'

const safeJSON = JSONBig({ useNativeBigInt: true })

interface MyRecord {
  a: number;
  b: bigint;
}

const example1 = '{ "a": 14, "b": 42 }'
const example2 = '{ "a": 14, "b": 43543763985612362153 }'

const record1 = safeJSON.parse(example1) as MyRecord
const record2 = safeJSON.parse(example2) as MyRecord

console.log(typeof record1.b)
// number

console.log(typeof record2.b)
// bigint

/* This one is commented-out because it is a (legit) compilation error. */ 
// console.log(record2.b + record1.a) 
//
//     TSError: ⨯ Unable to compile TypeScript:
//     tmp.ts:22:13 - error TS2365: Operator '+' cannot be applied to types 'bigint' and 'number'.

/* This one compiles, but gives a runtime error because the _real_ underlying types of `b` in both records are different! */
console.log(record1.b + record2.b)
// console.log(record1.b + record2.b)
//                      ^
// TypeError: Cannot mix BigInt and other types, use explicit conversions

Turning { alwaysParseAsBig: true } does not help at all, because now every single number get parsed as a bigint which is also an unwanted behavior. Ideally, I wish the library could use bigint if the resulting type is a bigint... I don't know however if this is possible in TypeScript? It is pretty common (and powerful) in Haskell or Elm to write functions as such:

parse :: ByteString -> Maybe a

where a is really instantiated by the caller, and triggers instance resolutions and other appropriate lookups on the compiler.

[KtorZ]

The only "okay-ish" option I see is to not declare types as bigint but as number | bigint, and then, let the call-site handle it properly. Alternatively, we could do a second pass on the parsed-data to make sure that any type declared as bigint ends up being deserialised as a bigint... but I am not sure how far we can go with generics and reflection in TypeScript.

[KtorZ]

Seems like there's close to no support for runtime type reflection in TypeScript.. I've found: https://www.npmjs.com/package/reflec-ts which gets us into a slippery slope I am not a fond of. So it seems like the number | bigint is perhaps the best compromise. It doesn't concern many types anyway, so the annoyance is limited, but still ...

For example, this works:

import JSONBig from 'json-bigint'

const safeJSON = JSONBig({ useNativeBigInt: true })

interface MyRecord {
  a: number;
  b: number | bigint;
}

const example1 = '{ "a": 14, "b": 42 }'
const example2 = '{ "a": 14, "b": 43543763985612362153 }'

const record1 = safeJSON.parse(example1) as MyRecord
const record2 = safeJSON.parse(example2) as MyRecord

function sum (a: number | bigint, b: number | bigint) : number | bigint {
  if (typeof a === 'number' && typeof b === 'number') {
    return a + b
  }

  if (typeof a === 'number' && typeof b === 'bigint') {
    return BigInt(a) + b
  }

  if (typeof a === 'bigint' && typeof b === 'number') {
    return a + BigInt(b)
  }

  return (a as bigint) + (b as bigint)
}

console.log(typeof record1.b)
// number

console.log(typeof record2.b)
// bigint

console.log(sum(record2.b, record1.a))
// 43543763985612362167n

console.log(sum(record1.b, record2.b))
// 43543763985612362195n

[KtorZ]

Some idea (not my most elegant code...), write a little wrapper around the Json parser to sanitize a parsed result, based on some whitelisting. In the end, there aren't many interfaces which require bigints (namely, assets and metadatum) thus it could work like this:

https://gist.github.com/KtorZ/010b8ea3e703cc669f762635c912c959

The drawbacks obviously is a performance hit for every parsed results which needs to be traversed an additional time, and, it completely escapes the compiler's watch. If we add new interfaces which require bigint, we need to remember to add them there. On the plus side, it's transparent and neat for the library users.

[rhyslbw]

Agree with the assessment here, and your solution is very reasonable solution given the constraints. Without the coercion handled internally it's a complex mess for implementations to handle.

[KtorZ]

So, I ended up implementing this for now: 84cf365

(not fully tested yet though...)

Which I find only mildly satisfactory. Yet, it should get us going for the time being and, we have to remember being careful about possible new bigint types. In the meantime, I had a look at the json-bigint library which indeed seems a bit abandoned. Beside our little issue here, I found at least two other issues which are worth fixing:

• The decision for turning a number into a bigint is currently broken on json-bigint. There's a fixed version of the library re-uploaded as true-json-bigint. The fix is really trivial though (using toFixed() in one place).

• There's PR#47 which adds some interesting option / support to have an Object prototype on the parsed object. I am not sure why the implementor went for a null prototype initially, this does not seem anecdotal, but maybe done for wrong reason. Anyway, having resulting object with null prototype is a bit cumbersome as typical functions on objects aren't available :| ... We had a failing test because of that.

So, all-in-all, I think it's worth forking the library anyway if only for these two changes. Then, I've started working on an implementation which would give us an API like:

  const _1 = JSONBig.parse(
    '{ "foo_1": 14, "foo_2": 42 }',
    schema.properties["Foo"]
  )
  console.log(_1);
  // { foo_1: 14n, foo_2: 42 }

  const _2 = JSONBig.parse(
    '[{ "bar": 42, "foo": 42 }, { "bar": 14, "foo": 14 }]',
    schema.properties["Bar"]
  )
  console.log(_2);
  // [ { bar: 42, foo: 42n }, { bar: 14, foo: 14n } ]

which I find pretty neat. Since we know the expected schema when parsing, this would be quite easy to implement. However, my current hack does not handle the compound combinators well yet (anyOf, oneOf & allOf) which are quite challenging to handle correctly (since they can potentially lead to a choice between multiple candidate schemas). So, I'd be happy to continue working on that as a separate stream of work, while we still get going with the current fix and the post-processing. What do you think @rhyslbw ?

[rhyslbw]

I'm happy with this result and agree we should decouple the optimisations.