BadgerDB does not reclaim disk space after pruning

[martintomazic]

Update:
Stopping the node, to enable pruning or to configure less blocks to keep "works" in a sense that clients will not see old blocks or be able to query old states.

However, it will not release disk space of existing badgerDB instances.

[martintomazic]

From state sync experience, I can only say that blocks are not pruned but state is. And even if you prune blocks manually, badger doesn't shrink space. So once you finished sync, you are probably screwed. When needed, I locally prune blocks, copy db and replace it then.

As from today call on the product demo by @peternose

I can confirm the same. Did a block sync from the genesis, and few days later (after being synced) my blockstore is still stable at 279GB. The last retained height is trimmed correctly (node status), consensus client is also unable to fetch old blocks (as expected), however what stays inside badger is a different story.

I tried checking the cometbft code, logs etc. and generally it all seems sound to me. As a final resort I was planing to patch the code (cometbft) and count all pruned height and make sure there is no bug, e.g. possible issue with updating the base incorrectly in their store.go.

But my understanding this is not needed as you have already figured the issue?

And even if you prune blocks manually, badger doesn't shrink space.

But after a while it should or? If you could elaborate here that would be awesome! :)

cc @ptrus also.

[peternose]

You can try to open the database to see how many blocks are there. Then, you could copy the database using the provided code without pruning the blocks.

package main

import (
	"fmt"
	"log"
	"path/filepath"

	cmtconfig "github.com/cometbft/cometbft/config"
	cmtnode "github.com/cometbft/cometbft/node"
	"github.com/cometbft/cometbft/store"
	"github.com/dgraph-io/badger/v4"
	"github.com/dgraph-io/badger/v4/options"
	"github.com/spf13/viper"

	tmcommon "github.com/oasisprotocol/oasis-core/go/consensus/cometbft/common"
	"github.com/oasisprotocol/oasis-core/go/consensus/cometbft/db"
)

func main() {
	dataDir := "mainnet/data"
	keepN := 100

	pruneDB(dataDir, keepN)
	copyDB(dataDir)
}

func pruneDB(dataDir string, keepLastN int) error {
	dbProvider, err := db.GetProvider()
	if err != nil {
		return fmt.Errorf("failed to get DB provider: %w", err)
	}

	cmtConfig := cmtconfig.DefaultConfig()
	if err := viper.Unmarshal(&cmtConfig); err != nil {
		return fmt.Errorf("failed to unmarshal config: %w", err)
	}
	cmtConfig.SetRoot(filepath.Join(dataDir, tmcommon.StateDir))

	blockStoreDB, err := dbProvider(&cmtnode.DBContext{ID: "blockstore", Config: cmtConfig})
	if err != nil {
		return fmt.Errorf("failed to open blockstore DB: %w", err)
	}
	defer blockStoreDB.Close()

	store := store.NewBlockStore(blockStoreDB)

	fmt.Printf("Block store size before pruning: %d (height=%d)\n", store.Size(), store.Height())

	pruneFrom := store.Height() - int64(keepLastN) + 1
	pruned, err := store.PruneBlocks(pruneFrom)
	if err != nil {
		return fmt.Errorf("failed to prune blocks: %w", err)
	}

	fmt.Printf("Pruned %d blocks. New size: %d (height=%d)\n", pruned, store.Size(), store.Height())

	return nil
}

func openBadger(path string) (*badger.DB, error) {
	opts := badger.DefaultOptions(path).
		WithSyncWrites(false).
		WithCompression(options.Snappy).
		WithBlockCacheSize(64 * 1024 * 1024)
	return badger.Open(opts)
}

func copyDB(dataDir string) {
	srcPath := fmt.Sprintf("%s/consensus/data/blockstore.badger.db", dataDir)
	dstPath := fmt.Sprintf("%s/consensus/data/blockstore2.badger.db", dataDir)

	dbSrc, err := openBadger(srcPath)
	if err != nil {
		log.Fatalf("failed to open source DB: %v", err)
	}
	defer dbSrc.Close()

	dbDst, err := openBadger(dstPath)
	if err != nil {
		log.Fatalf("failed to open destination DB: %v", err)
	}
	defer dbDst.Close()

	fmt.Println("Copying data...")

	writeBatch := dbDst.NewWriteBatch()
	defer writeBatch.Cancel()

	err = dbSrc.View(func(txn *badger.Txn) error {
		it := txn.NewIterator(badger.DefaultIteratorOptions)
		defer it.Close()

		for it.Rewind(); it.Valid(); it.Next() {
			item := it.Item()
			k := item.KeyCopy(nil)
			val, err := item.ValueCopy(nil)
			if err != nil {
				return err
			}

			if err := writeBatch.Set(k, val); err != nil {
				return err
			}
		}
		return nil
	})
	if err != nil {
		log.Fatalf("failed to copy data: %v", err)
	}

	if err := writeBatch.Flush(); err != nil {
		log.Fatalf("failed to flush batch: %v", err)
	}

	fmt.Println("Copying ended.")
}

[martintomazic]

You can try to open the database to see how many blocks are there.

I used your script and immediately 279GB was transformed into 70 MB. Furthermore, did some additional sanity checks...

In practice this means that badger has this keys marked as non-existent, but they are still pending removal, likely nested deep in the SSTables, still using space.

I find it strange that even after few days of running the node (new blocks coming every 6 seconds), the badger is still unable to reclaim the space likely due to the pending compaction? Which is strange given that new block every 6 seconds should definitely not overload the system?

Probably I should check/experiment with the badger config, especially things related to compaction?

Finally, we could also consider exporting the script you provided as a node util for the maintenance...

[martintomazic]

cc @anzoman for visibility/to chip in, given that you have been experimenting with this on the devops side already.

[anzoman]

Thanks for looking more into this @martintomazic. For us, we had a Mainnet observer node with full history for Consensus and all runtimes, and we started running out of disk. The experience was that pruning started working for both, but looking from the logs, we saw it pruned like one day of blocks per one day of running, and the nodes also became unhealthy as the runtimes could not keep up with the new blocks. CPU usage was maxed out (and we did not have available resources to increase it).

So, we also tried disabling runtimes and runtime punning, and the results were not so much better. In 27 hours, it pruned a little less than 2 days of blocks. The resources were no longer maxed out, but since that pruning happens in intervals, it was not fast enough for us. There were also no changes in the overall machine disk usage (no disk space was freed).

Since we would run out of disk before pruning can release disk space, we then just decided go with state sync.

[martintomazic]

The experience was that pruning started working for both, but looking from the logs, we saw it pruned like one day of blocks per one day of running, and the nodes also became unhealthy as the runtimes could not keep up with the new blocks. CPU usage was maxed out (and we did not have available resources to increase it).

I see. Looking from your logs in https://github.com/oasisprotocol/internal-ops/issues/962, this observation relates to state pruning, i.e. it is slow and uses lots of resources (makes sense given mkvs). Update: Looking at logs again, this was because pruning already finished, thus the rate of new blocks is the rate of the pruning.

So this is orthogonal to the issue described above. Likely the only way to fix it is either prune from the start, or increase the resources during the prune part, which is not possible on the bare-metal instances. So probably there is not way to fix this unless making underlying nodedb more efficient.

Blockstore on the other hand gets pruned almost instantaneously, from the client perspective. Unfortunately inside badgerdb it is a different story as explained above.

Update: Both blockstore and state (mkvs) get pruned in less then 1h. The disk space however is not reclaimed and stays stable after the pruning is enabled.

[peternose]

Manually pruning the block store and calling Flatten also works, as compactions are forced, so no need to copy the store.

[martintomazic]

Manually pruning the block store and calling Flatten also works, as compactions are forced, so no need to copy the store.

Nice!

This solves an existing issue of Cometbft only starting the blockstore pruning once the node is synced (#6319)

Since as per documentation, there should be no writes, this could be done by pausing the block sync and calling prune and flatten procedure sporadically. Alternatively this could be solved on the infra side, by restarting the node few times during long block sync, and calling the new command as proposed (#6309).

The only thing still pending a fix is the scenario for the validator node, where we cannot have downtime, so pausing/restarting a node to trigger a manual compaction is not an option.

[martintomazic]

I think badger does not reclaim blockstore space as given its logs it thinks it has nothing to reclaim. Thus compaction (at least of the bottom level) is never triggered.

{"caller":"helpers.go:41","level":"info","module":"cometbft/db/badger","msg":"Level 0 [ ]: NumTables: 03. Size: 118 MiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 2 [ ]: NumTables: 00. Size: 0 B of 28 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 4.0 MiB\nLevel 3 [ ]: NumTables: 00. Size: 0 B of 282 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB\nLevel 4 [ ]: NumTables: 00. Size: 0 B of 2.8 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 16 MiB\nLevel 5 [B]: NumTables: 531. Size: 1.0 GiB of 28 GiB. Score: 0.00->0.00 StaleData: 1.6 GiB Target FileSize: 32 MiB\nLevel 6 [ ]: NumTables: 4707. Size: 276 GiB of 276 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel Done","path":"/node/data/consensus/data/blockstore.badger.db","ts":"2025-09-01T12:29:15.153608695Z"}

Level 6 [ ]: NumTables: 4707. Size: 276 GiB of 276 GiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel

For contrast mkvs/pathbadger has some stale data marked in the bottom level (thus disk space is eventually released):

{"caller":"helpers.go:41","level":"info","module":"mkvs/db/pathbadger","msg":"Level 0 [ ]: NumTables: 03. Size: 85 MiB of 0 B. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 64 MiB\nLevel 1 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 2 [ ]: NumTables: 00. Size: 0 B of 10 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 2.0 MiB\nLevel 3 [ ]: NumTables: 00. Size: 0 B of 17 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 4.0 MiB\nLevel 4 [ ]: NumTables: 00. Size: 0 B of 174 MiB. Score: 0.00->0.00 StaleData: 0 B Target FileSize: 8.0 MiB\nLevel 5 [B]: NumTables: 154. Size: 1.7 GiB of 1.7 GiB. Score: 0.00->0.00 StaleData: 484 MiB Target FileSize: 16 MiB\nLevel 6 [ ]: NumTables: 666. Size: 17 GiB of 17 GiB. Score: 0.00->0.00 StaleData: 1.0 GiB Target FileSize: 32 MiB\nLevel Done","ts":"2025-09-01T12:29:18.230463128Z"}

• Level 6 [ ]: NumTables: 666. Size: 17 GiB of 17 GiB. Score: 0.00->0.00 StaleData: 1.0 GiB Target FileSize: 32 MiB