Revamp VC key loading

[syjn99]

We have several points in VC key loading that can be improved.

Problem 1: Fake wallet for Web3Signer

prysm/validator/accounts/wallet/wallet.go

Lines 255 to 265 in f9c7a8d

	// NewWalletForWeb3Signer returns a new wallet for web3 signer which is temporary and not stored locally.
	func NewWalletForWeb3Signer(cliCtx *cli.Context) *Wallet {
	walletDir := cliCtx.String(flags.WalletDirFlag.Name)
	// wallet is just a temporary wallet for web3 signer used to call initialize keymanager.
	return &Wallet{
	walletDir: walletDir, // it's ok if there's an existing wallet
	accountsPath: "",
	keymanagerKind: keymanager.Web3Signer,
	walletPassword: "",
	}
	}

Remote signing doesn't need keys on disk, but we simply add a "stub" wallet only to satisfy the interface. Decouple Web3Signer with the current Prysm wallet design.

Problem 2: Wallet registration should be preceded for Keymanager API

walletInitialized is the single gate that guards the Keymanager API handlers like POST /eth/v1/keystores. We can remove the blocker regardless of wallet initialization, as it is a normal workflow that users can POST keystores after running VC process.

Problem 3: Non-standard format & No direct key loading

Every other clients (Lighthouse, Lodestar) use per-key EIP-2335 files. This makes users harder to migrate from other VC to Prysm VC due to compatability issue. We can add one more "store" implementation apart from the current Prysm wallet. So the desired flags that we can have are --validator-keys and --keystore-passwords.

[syjn99]

Rollout Plan

Note

• Deprecate interop related flags in VC #17163

Removing interop flags will probably make our lives easier; As this deletes one possible path for loading VC keys.

See full branch (mostly built by Fable but should be okay to explore the feasibility of this plan) below:

• Rollout for adding EIP-2335 path for Prysm VC key loading syjn99/prysm#22

1. Web3Signer keymanager without a wallet

• Commit: syjn99@856a3bb (Construct the web3signer keymanager without a wallet)
• Key change: Remove creating a fake wallet for Web3Signer, and fix walletInitialized condition to
  • cfg.Wallet != nil || (cfg.ValidatorService != nil && cfg.ValidatorService.RemoteSignerConfig() != nil)
• This resolves Problem 1. Fake wallet for Web3Signer - doesn't touch any other path so risk should be low.

2. Add an abstraction layer that can accommodate both legacy wallet path and new EIP-2335 path

• Commit A: syjn99@9e15884 (Introduce AccountStore with wallet and directory backends)
• Commit B: syjn99@e25a8b3 (Add --validator-keys and --keystore-passwords for direct keystore loading)
• Key change: Add a new abstraction interface -

type AccountStore interface {
	// Load reads and decrypts all stored accounts.
	Load(ctx context.Context) (*accountStore, error)
	// Save persists the full account set.
	Save(ctx context.Context, store *accountStore, keystores []*keymanager.Keystore, passwords []string) (bool, error)
	// WatchPath is the file or directory watched for external changes.
	// Consumed by listenForAccountChanges.
	WatchPath() string
}

• ...and implement for Prysm wallet and EIP-2335 key files.
• This resolves Problem 3: Non-standard format & No direct key loading - also not touching the existing path but adding a new option for users to use EIP-2335 keystore. EIP-2335 keystore can be brought from 1) staking-deposit-cli or 2) other VC clients.

3. Walletless Keymanager API + Remove Web UI Support

• Commit: syjn99@803ec21 (Make the keymanager API walletless and remove the frozen web UI)
• Key change: Delete walletInitializedFeed so that --rpc path can be initialized via Keymanager API. Also delete web path for simplicity.
• This resolves Problem 2: Wallet registration should be preceded for Keymanager API.

4. Rewire the accounts CLI - groundwork for deprecating the wallet

• Commit: syjn99@079b116 (Wire the accounts CLI to --validator-keys and deprecate the wallet)
• Key change: validator accounts import/list/delete/backup/voluntary-exit now resolve their key source through EIP-2335 keystores when --validator-keys are set, otherwise the legacy wallet. Print deprecation log when a new user wants to use validator wallet create|recover, and --wallet-dir flag.
• This provides us few releases window for users to be prepared. Later on we can fully delete wallet path in the next (or next-next) major release.