[Benchmark] - Store pins in a datastore instead of a DAG #7674
Description
Recently, JS-IPFS recently merged in a PR that switched the pinset to being stored in a datastore instead of a DAG (ipfs/js-ipfs#2771).
Go-IPFS could benefit from the same update being made to how it stores pinsets as pinning becomes drastically slower when working with larger pinsets.
There are a few graphs in the PR linked above that show how vastly this improved pinning / unpinning times for js-ipfs. In addition, below are some statistics from just a few of the nodes Pinata is running around the world. As can be seen, this impacts both flatFS and BadgerDB.
| Node Name | FileSystem | Number of root CIDs Pinned | Average ms to Pin of < 100KB file | Average ms to Unpin Time of < 100KB file |
|---|---|---|---|---|
| Fresh Local Node | FlatFS | 5 | 20 | 12 |
| Node-1 | FlatFS | 112300 | 444 | 425 |
| Node-2 | FlatFS | 304738 | 1311 | 1301 |
| Node-3 | FlatFS | 465270 | 1696 | 1779 |
| Node-4 | FlatFS | 4686 | 37 | 38 |
| Node-5 | FlatFS | 5836 | 79 | 67 |
| Node-6 | FlatFS | 5609 | 82 | 54 |
| BadgerNode-1 | BadgerDB | 18558 | 151 | 145 |
| BadgerNode-2 | BadgerDB | 16667 | 113 | 108 |
| BadgerNode-1 (2 days later) | BadgerDB | 28627 | 180 | 177 |
| BadgerNode-2 (2 days later) | BadgerDB | 26806 | 222 | 193 |
In addition to this, it appears that if there are multiple pins being added to the DAG pinset at the same time, each of these update will have to wait in line to be updated sequentially, which has the potential to create very long backups if a node gets a lot of traffic at once.
Related Issues: