Big file parsing

[GDreyV]

I have prettified json file over 110Mb which is relatively big but not huge. I read it into string for less than 1 second and then call json::parse which takes 225 seconds and over 500Mb of memory. Am I doing something wrong? Why is it that slow and consumes such amount of memory?

[nlohmann]

The DOM parser builds the complete JSON object into memory. If you just want to do a syntax check or want only perform a few specific operations, you should use the SAX parser. What is your usecase?

[GDreyV]

I need to iterate over json elements so I need entire DOM. It seems that parser increases document size in 10+ times, in my case original minified json is about 40Mb.

[nlohmann]

A third option could be the callback function with which you can work on the data yet still decide whether you want to store them for later use.

[GDreyV]

I mean just regular nodejs json parse takes about 2 seconds for the same file.

[nlohmann]

I would be interested in the file to understand where we take so much time.

[GDreyV]

Sure, here's a small repo to reproduce the flow.
UPD: Sorry, I had to drop the repo as file might include sensitive data. I will mail the repo to you, please do not publish it.

[GDreyV]

Sorry, it was my fault, it was test with enabled memory corruption limits. Real execution time is:
2399 milliseconds to read json
3193 milliseconds to read and dispose json
And memory usage is about 130Mb on 45Mb data file.