Skip to content

Data queue memory leak itoolkit 0.1.6 + idb-connector #230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bmcq92 opened this issue Apr 6, 2020 · 3 comments
Closed

Data queue memory leak itoolkit 0.1.6 + idb-connector #230

bmcq92 opened this issue Apr 6, 2020 · 3 comments
Labels
bug Something isn't working

Comments

@bmcq92
Copy link

bmcq92 commented Apr 6, 2020

We're having issues in production similar to this issue raised in idb-connector a while ago.

We're seeing the temporary storage use on the node job increase rather fast when reading and writing to data queues. This doesn't happen when using the REST transport so I thought this may be an issue with idb-connector. On further testing, I've found the issue doesn't seem to occur with the newer alpha1 version of itoolkit in npm repos using the Toolkit class to read/write data queues so decided to raise it here instead.

Are you still supporting issues with the older version of itoolkit? If not, are there plans to implement reading/writing keyed data queues in the newer version soon™ (it looks like Toolkit dataqueue stuff is being canned)?

Thanks.

@bmcq92 bmcq92 added the bug Something isn't working label Apr 6, 2020
@kadler
Copy link
Member

kadler commented Apr 6, 2020

There are some big overhauls going on, though we plan to be 99% compatible with v0.1 in v1.0. Those APIs will be deprecated once #151 lands, but they will stick around for the whole v1 lifetime. Please open a feature request if you want keyed data queue support.

As to this issue, looks like were not closing the statement object we create. Compare v1.0.6 with master

@ThePrez
Copy link
Member

ThePrez commented Apr 6, 2020

Also of note, the new DATA_QUEUE_* SQL services have support for keyed data queues and allow you to read/write.

kadler added a commit that referenced this issue Apr 6, 2020
Deleting the statement object doesn't free the underlying resources,
causing memory leaks. Instead, we need to close both the connection and
the statement to free these resources. We can let the runtime delete the
objects as well.

Fixes #230
@bmcq92
Copy link
Author

bmcq92 commented Apr 6, 2020

Thanks for the quick response. Tested your changes above and it looks much better now. Job temporary storage use is down at 50MB after 1000 send and receives rather than in the 300's and increasing.

Will raise a feature request for keyed data queues tomorrow.

@bmcq92 bmcq92 closed this as completed Apr 6, 2020
kadler added a commit that referenced this issue Apr 9, 2020
Deleting the statement object doesn't free the underlying resources,
causing memory leaks. Instead, we need to close both the connection and
the statement to free these resources. We can let the runtime delete the
objects as well.

Fixes #230
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants