How to clear memory after writing to a file within a loop

[ThomasTJdev]

When writing multiple PDF files within a loop the memory consumption just steadily increases. Is there a way to clear or free memory after each write?

Example

import nimPDF/nimPDF

var opts = newPDFOptions()
opts.addImagesPath("pic")
var doc = newPDF(opts)

for i in 0..10:
  doc.addPage(getSizeFromName("A4"), PGO_PORTRAIT)
  doc.drawText(15, 15, "Hello World!")
  
  var image = doc.loadImage("large-pic.jpg")
  var x = 40.0
  var y = 40.0
  doc.drawImage(x, y, image)
 
  if not doc.writePDF("hello.pdf"):
    echo "cannot open: hello.pdf"
 
  echo "------------------"
  GC_fullCollect()
  echo GC_getStatistics()

  # Can I free the memory here?
 
  doc = newPDF(opts)

Output

before it's pic/e.jpg falling back!
------------------
[GC] total memory: 495083520
[GC] occupied memory: 109322832
[GC] stack scans: 341
[GC] stack cells: 88
[GC] cycle collections: 3
[GC] max threshold: 327681600
[GC] zct capacity: 2304
[GC] max cycle table size: 0
[GC] max pause time [ms]: 0
[GC] max stack size: 4768

before it's pic/e.jpg falling back!
------------------
[GC] total memory: 569765888
[GC] occupied memory: 109323616
[GC] stack scans: 690
[GC] stack cells: 90
[GC] cycle collections: 4
[GC] max threshold: 327681600
[GC] zct capacity: 2304
[GC] max cycle table size: 0
[GC] max pause time [ms]: 0
[GC] max stack size: 4768

before it's pic/e.jpg falling back!
------------------
[GC] total memory: 569765888
[GC] occupied memory: 109323552
[GC] stack scans: 1039
[GC] stack cells: 90
[GC] cycle collections: 5
[GC] max threshold: 327681600
[GC] zct capacity: 2304
[GC] max cycle table size: 0
[GC] max pause time [ms]: 0
[GC] max stack size: 4768

before it's pic/e.jpg falling back!
------------------
[GC] total memory: 650440704
[GC] occupied memory: 109323616
[GC] stack scans: 1388
[GC] stack cells: 90
[GC] cycle collections: 6
[GC] max threshold: 327681600
[GC] zct capacity: 2304
[GC] max cycle table size: 0
[GC] max pause time [ms]: 0
[GC] max stack size: 4768

xxxxx

[jangko]

isn't the occupied memory relative stable? looks like nim is a bit lazy to return the memory page to the OS. this is beyond app control.

[ThomasTJdev]

When writing large files the memory expands quite a lot. I have some PDF files which ranges between 150-500mb, and then the memory usage goes from 500mb to +16GB.
With large files like this it is really clear, that each loop just appends new memory without any cleanup - the filewriting memory usage is never released.

[ThomasTJdev]

Here is real example with 4 files increasing up to 5.6GB.

# Loop1 - 425mb filesize
[GC] occupied memory: 648737024
[GC] total memory: 810221568

# Loop2 - 423mb filesize
[GC] occupied memory: 1941934384
[GC] total memory: 2394058752

# Loop3 - 435mb filesize
[GC] occupied memory: 2865117408
[GC] total memory: 3682131968

# Loop4 - 1.2gb filesize
[GC] occupied memory: 2506910752
[GC] total memory: 5641519104

------


+------+--+---------------+----------------+--+-------------------+----------------+--+-------------+
| Loop |  | Occupied (mb) | Diff from prev |  | Total memory (mb) | Diff from prev |  | Filesize MB |
+------+--+---------------+----------------+--+-------------------+----------------+--+-------------+
|    1 |  |           649 |              0 |  |               810 |              0 |  |         425 |
|    2 |  |          1942 |           1293 |  |              2394 |           1584 |  |         423 |
|    3 |  |          2865 |            923 |  |              3682 |           1288 |  |         435 |
|    4 |  |          2507 |           -358 |  |              5642 |           1959 |  |        1200 |
+------+--+---------------+----------------+--+-------------------+----------------+--+-------------+

[jangko]

well, perhaps you can report this issue to nim repo. I don't know how can I help you. I also expect nim to automatically do garbage collection. why would it become our concern. I really cannot help you.

[ThomasTJdev]

Check, I'll move it to the nim-repo. I just wasn't sure if there was a way to manually free PDF*.