Fix "OldNonce" errors in Request Node #1292
Description
Problem
Intermittently, the Gnosis Gateway emits errors like the following:
Recent errors
Mar 12 21:40:42 prod gnosis_request-node.1.q1xjvoqo9lndwfso7ma0lq236[797]: [21:40:42] ERROR: persistTransaction fail {"error":{"reason":"nonce has already been used","code":"NONCE_EXPIRED","error":{"reason":"processing response error","code":"SERVER_ERROR","body":"{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32010,\"message\":\"nonce too low\"},\"id\":67}","error":{"code":-32010},"requestBody":"{\"method\":\"eth_sendRawTransaction\",\"params\":[\"0x02f9015a64829a448477359400850218711a0082d36f94268c146afb4790902ee26a6d2d3aff968623ec80880162ea854d0fc000b8e451a58abb000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000002e516d576770356b526d6f675a746f5953694a77666b786b69785263524d50533245396f465969396379764d48664700000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000788c080a0311b046f23959c657f5e966355ef41d3a7cca40e93023cbbc236ad212f18d7e4a051b1ca0fe2a9c8c02f6ff2c0e1113caf7b9f3aba04b038f26644e6a752c90917\"],\"id\":67,\"jsonrpc\":\"2.0\"}","requestMethod":"POST","url":"node RPC omitted for privacy"},"method":"sendTransaction","transaction":{"type":2,"chainId":100,"nonce":39492,"maxPriorityFeePerGas":{"type":"BigNumber","hex":"0x77359400"},"maxFeePerGas":{"type":"BigNumber","hex":"0x0218711a00"},"gasPrice":null,"gasLimit":{"type":"BigNumber","hex":"0xd36f"},"to":"0x268C146Afb4790902Ee26A6D2d3aff968623Ec80","value":{"type":"BigNumber","hex":"0x0162ea854d0fc000"},"data":"0x51a58abb000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000002e516d576770356b526d6f675a746f5953694a77666b786b69785263524d50533245396f465969396379764d48664700000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000788","accessList":[],"hash":"0x4a9c944a4eac979a6fd0e83e3ca19d6b0134b191ad28acff5921515b4d862aec","v":0,"r":"0x311b046f23959c657f5e966355ef41d3a7cca40e93023cbbc236ad212f18d7e4","s":"0x51b1ca0fe2a9c8c02f6ff2c0e1113caf7b9f3aba04b038f26644e6a752c90917","from":"0xe76b7c6BC7c596D9c8c7ee2b99a6338cDd4A2D00","confirmations":0},"transactionHash":"0x4a9c944a4eac979a6fd0e83e3ca19d6b0134b191ad28acff5921515b4d862aec"},"transactionHash":{"type":"keccak256","value":"0xbccfc1508b2392624bf03f3f418f09e631c63d53dc0f85d2988ed9d3ae4c2c83"},"channelId":"011b255b9fc764fbd538a0634ed7a3a51264a4e54c25b3752cf19bce485dc3b268","topics":["012c4f7cc43b0fc8389d59f0c00588321b979c2703b68d967d66a2524f2da7f896","0147d73aa38da26d5de1e0b47e1e517c035c57f5f8fe61ca70cb61198238a7fa09","011b255b9fc764fbd538a0634ed7a3a51264a4e54c25b3752cf19bce485dc3b268"],"transactionData":{"request contents omitted for privacy"}}
Past Errors
[14:56:07] ERROR: persistTransaction error: Error: processing response error (body="{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32010,\"message\":\"OldNonce, Current nonce: 84465, nonce of rejected tx: 84460\"},\"id\":65}", error={"code":-32010}, requestBody="{\"method\":\"eth_sendRawTransaction\",\"params\":[\"0x02f9015b64830149ec84d473b848850539ebb04682d08f94268c146afb4790902ee26a6d2d3aff968623ec80880162ea854d0fc000b8e451a58abb000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000002e516d6163615552586e463652614e796d3650574d64466e54384831455a3651725757446d6765696456484e4a5139000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000005aec080a0bf7f53282914e4df9554ebadc9490fcdaf9a562a9d998cc90f791d27620d4141a043a7e8490d69fefae47b8c5d989361192939f341571af6dabc7283b126b6ae8e\"],\"id\":65,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="<node RPC omitted for privacy>", code=SERVER_ERROR, version=web/5.5.0)
When these errors occur, the /persistTransaction endpoint in the Request Node will fail to persist new requests in Request Network - it fails to submit the blockchain transaction to our storage smart contract on Gnosis - because the nonce is lower than the current nonce.
We believe the root cause of this error is the NonceManager, an experimental feature from Ethers v5.
The current implementation of the NonceManager in ethers.js does not handle errors. If the RPC refuses the transaction the NonceManager will still increment the nonce, so consecutive transactions are not processed, see https://github.com/ethers-io/ethers.js/blob/6ee1a5f8bb38ec31fa84c00aae7f091e1d3d6837/src.ts/providers/signer-noncemanager.ts#L83
Bandaid Solution
Whenever these errors occur, the solution has been to restart the Request Node.
Unfortunately, since the pendingTransaction cache is stored in Request Node memory, all pending requests are dropped. This behavior is scheulded to be fixed in:
Fixing this issue may also fix #1479 depending on how it's implemented.
Proposed Solution
- Replace the
NonceManagerwith a fault-tolerant alternative.
History
Attempts have been made to fix this error, but they were unsuccessful.