[feature]: resolve inadvertent 1st party universe tree collision mapping

[Roasbeef]

Is your feature request related to a problem? Please describe.

Today we have an issue where an issuance or transfer event can result in a non-unique Universe key, which can result in some data being lost/clobbered. It's important to note that due to the key structure here, only the sender or transaction creator can trigger this behavior.

To understand the issue, we need to look at the Universe tree structure. For simplicltiy, we'll focus on issuance events here.

For assets that have a group key, the top level key is a special universe ID that's just the group key:

taproot-assets/universe/interface.go

Lines 55 to 76 in 071c1d0

	// Identifier is the identifier for a universe.
	type Identifier struct {
	// AssetID is the asset ID for the universe.
	//
	// TODO(roasbeef): make both pointers?
	AssetID asset.ID
	// GroupKey is the group key for the universe.
	GroupKey *btcec.PublicKey
	// ProofType is the type of proof that should be stored in the universe.
	ProofType ProofType
	}
	// Bytes returns a bytes representation of the ID.
	func (i *Identifier) Bytes() [32]byte {
	if i.GroupKey != nil {
	return sha256.Sum256(schnorr.SerializePubKey(i.GroupKey))
	}
	return i.AssetID
	}

This is used to derive the unique Universe namespace where we store the information on disk.

The leaf key then has the following structure:

taproot-assets/universe/interface.go

Lines 262 to 290 in 071c1d0

	// LeafKey is the top level leaf key for a universe. This will be used to key
	// into a universe's MS-SMT data structure. The final serialized key is:
	// sha256(mintingOutpoint || scriptKey). This ensures that all
	// leaves for a given asset will be uniquely keyed in the universe tree.
	type LeafKey struct {
	// OutPoint is the outpoint at which the asset referenced by this key
	// resides.
	OutPoint wire.OutPoint
	// ScriptKey is the script key of the base asset. If this isn't
	// specified, then the caller is attempting to query for all the script
	// keys at that minting outpoint.
	ScriptKey *asset.ScriptKey
	// TODO(roasbeef): add asset type too?
	}
	// UniverseKey is the key for a universe.
	func (b LeafKey) UniverseKey() [32]byte {
	// key = sha256(mintingOutpoint || scriptKey)
	h := sha256.New()
	_ = wire.WriteOutPoint(h, 0, 0, &b.OutPoint)
	h.Write(schnorr.SerializePubKey(b.ScriptKey.PubKey))
	var k [32]byte
	copy(k[:], h.Sum(nil))
	return k
	}

The issue here pops up when a user issues a new grouped asset, and in the same transaction issues multiple new asset IDs with the same script key. The leaf tuple right now is (outpoint, scriptKey), so only one of those new asset UTXOs will be stored. Instead, we want a structure of (outpoint, assetID, scriptKey). This resembles the structure in the normal asset commitment, wherein the final level still commits to the asset ID for uniqueness purposes.

Describe the solution you'd like

We should fix this at two levels: the RPC service, and the DB schema itself.

First for the RPC service, we'll add a new version to the relevant set of RPC calls. As we'll be changing the leaf key here, we'll end up with a distinct MS-SMT root for the same set of data. The old version will query the old structure on disk, and the new version will use the updated key structure.

As for the DB, we can keep the exact same schema, but we'll start to mix a version into the universe.Identifier struct mentioned above. This will also be carried into the universe.LeafKey structure, where the version will also affect the final key created. V0 is today, v1 adds the asset ID into the leaf specifier.

The final step will be a migration. This migration will operate on the Go level, as we'll just have the code load the instance of the old tree(s), then insert all the leaf items into the new tree.

For the migration, we may want to take a look at this PR for golang-migrate which will enable us to run go code along side the migrations. We'll need to this this effectively to ensure we have the necessary amount of fault tolerance here. Either way, the migration code can be defensive (diff to minimize work needed), or just ignore any upsert errors, etc.

[ffranr]

Thin interface solution method: a3b0635#r2017767432

I think what we can do is make a slim interface that's just UniverseKey(), then we make a wrapper version that includes asset.ID, then update the function to use that.

[guggero]

Thin interface solution method: a3b0635#r2017767432

I think what we can do is make a slim interface that's just UniverseKey(), then we make a wrapper version that includes asset.ID, then update the function to use that.

This is only for new trees though. For existing ones, we need a way to deal with the fact that both old and new clients might query the universe at the same time, expecting different keys. So we need to duplicate the tree somehow, one with the old method for backward compatibility and a new one that uses the new key format.

[ffranr]

Thin interface solution method: a3b0635#r2017767432

I think what we can do is make a slim interface that's just UniverseKey(), then we make a wrapper version that includes asset.ID, then update the function to use that.

This is only for new trees though. For existing ones, we need a way to deal with the fact that both old and new clients might query the universe at the same time, expecting different keys. So we need to duplicate the tree somehow, one with the old method for backward compatibility and a new one that uses the new key format.

@guggero Yes, I agree. I think we should implement the same sort of versioning strategy that we used for the GroupKeyReveal upgrade from V0 to V1.

That would mean introducing a new interface, renaming universe.LeafKey to universe.LeafKeyV0, and adding a new universe.LeafKeyV1, rather than layering changes on top of a shared base. Might help keep the original V0 implementation clearly identifiable.

And then in the doc for V0 we can explain why it's been depreciated.

(From #1456 (comment))

As a start at least. And then we need to think about two trees.

[Roasbeef]

So we need to duplicate the tree somehow, one with the old method for backward compatibility and a new one that uses the new key format.

Yep. The route I outlined in the issue is a Go migration to read all the leaves, then insert into a new tree, but we pass a different implementation of universe.LeafKey.

It's also important to note that this is primarily an RPC level change, not a p2p/protocol change like the group key reveal (clients reject a proof if not aware of the new rules).

The way I outlined things above we:

1. Freeze the old tree structure by rejecting RPC clients that are using the older version (to be passed into the relevant RPCs).
   * We already have the outpoint and script key in universerpc.AssetKey, it just isn't being used as a key.
2. Do a Go migration to copy over all the relevant trees in place. A migration is used to add a new version field to the root MS-SMT pointers. This way we know what type of key to use when inserting/fetching.
3. Start to serve the new tree over RPC, only allowing insertions into this new tree.

For step 2, I think we can use the Copy method I add in this PR. It works in two phases:

• First gather up all the leaves as a map from key to leaf.
• Insert all the leaves into the target tree.

We can add an optional step at the first phase, where we transform the key we return based on the leaf value. In this specific case, we'd decode the leaf (sparse decode of proof) to extract the: outpoint, script key, and ID. Then we hash it using the new system.

Specific details re the final approach we arrive on for the Go migrations, I think the migration itself can be somewhat succinct.

[Roasbeef]

A recent PR added this struct which eaves the way to doing the migration I outlined above:

taproot-assets/universe/interface.go

Lines 277 to 286 in 1660f9c

	// UniqueLeafKey is an interface that allows us to obtain the universe key for a
	// leaf within a universe. This is used to uniquely identify a leaf within a
	// universe. Compared to LeafKey, it includes the asset ID of a leaf within the
	// universe key calculation.
	type UniqueLeafKey interface {
	LeafKey
	// LeafAssetID returns the asset ID for the leaf.
	LeafAssetID() asset.ID
	}