diff --git a/README.md b/README.md index 58994e9..3b0d339 100644 --- a/README.md +++ b/README.md @@ -1,62 +1,53 @@ # Apphashd -This is a framework for identifying the root cause of an appphash mismatch on cosmos based networks inspired by [apphash_calculator](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23) written by [freak12techno](https://github.com/freak12techno). - -The `apphashd` binary takes two inputs as arguments, absolute/relative paths to the `application.db` of nodes with opposing apphashes. It calculates and compares the apphashes of all the modules present in the db. It generates an array with the name of the modules which have differing hashes. - -This repo contains a script called `apphashd.sh` which is an end to end script which builds the `apphashd` binary, executes it and uses [iaviewer](https://github.com/cosmos/iavl/tree/master/cmd/iaviewer) to identify the message_type/data that caused the apphash to deviate on both the nodes. - -## Usage -Download and run the script -``` -wget https://raw.githubusercontent.com/vitwit/apphashd/main/apphashd.sh && chmod 755 apphashd.sh -./apphashd.sh -``` -The script creates a folder called `hashes` inside the `apphashd` folder which contains the module hashes of both the nodes, iavl trees of the differing modules, diff of iavl trees and the ascii decoded output of the diff which contains the root cause of apphash mismatch. - -### Example usage -This repo has a folder called `testdata` which contains the application.db of two nodes which were halted due to an apphash mismatch. We will use that to test the script. - -#### Download the testdata -``` +A tool to quickly identify the root cause of apphash mismatches on Cosmos-based networks. Inspired by [apphash_calculator](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23). + +## Features +- Compares two `application.db` directories from different nodes +- Calculates and compares module apphashes +- Identifies modules with differing hashes +- Uses [iaviewer](https://github.com/cosmos/iavl/tree/master/cmd/iaviewer) to pinpoint the exact data causing the mismatch + +## Quick Start +1. **Download and run the script:** + ```sh + wget https://raw.githubusercontent.com/vitwit/apphashd/main/apphashd.sh && chmod 755 apphashd.sh + ./apphashd.sh + ``` +2. **Results:** + - A `hashes` folder is created inside `apphashd`. + - Contains module hashes, IAVL trees, diffs, and decoded output showing the root cause of the mismatch. + +## Usage Example +This repo includes test data from two nodes with an apphash mismatch. + +### 1. Download test data +```sh mkdir ~/node1 ~/node2 cd ~/node1 wget https://github.com/vitwit/apphashd/raw/main/testdata/node1/node1-application-db.tar.gz -tar -xvf node1-application-db.tar.gz + tar -xvf node1-application-db.tar.gz cd ~/node2 wget https://github.com/vitwit/apphashd/raw/main/testdata/node2/node2-application-db.tar.gz tar -xvf node2-application-db.tar.gz ``` -#### Download and execute the script -``` +### 2. Run the script +```sh cd ~/ wget https://raw.githubusercontent.com/vitwit/apphashd/main/apphashd.sh && chmod 755 apphashd.sh ./apphashd.sh ~/node1/application.db ~/node2/application.db ``` -The script generated a file called `diff-bank-decoded` inside the `~/apphashd/hashes` folder. We can check the root cause of the apphash mismatch by inspecting this file. -``` -factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test - Ebzf>�����`Up6ɳk\vW�]Z��;Y -��6��N�.�v�X7m�;��&�\�ӻ�6Q -�X?R�%�����J��ğ5��'�І�qc�xv -�!9�E�ErB�l&��E��d^;��2��� -�@(x�Y�Bk5/RJ�jstake -woGϬ�U��ds*�;�+e�,ʣ^� -Q@�Sn��������n���)�H���3� -�@(x�Y�Bk5/RJ�jstake -��Mۗ�幗�t��pW�cU�N,��A�=jG�O -?,vD�t�S���Ѽ/lBq�+��cR��>n -��[h"%p�\0��ke���factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test -s;�ܝ"s�]����� shape-1.txt` - -Validator 2: -`iaviewer shape ~/.juno/data/application.db s/k:bank/ > shape-2.txt` - -**Note:-** the module name has to be mentioned in a specific format. I wanted to query the bank module so specified it as `s/k:bank/`. If any other module needs to be mentioned like `staking` or `slashing` it needs to be specified as `s/k:staking/`, `s/k:slashing/`etc. If the module name is not mentioned in that format then you'll see the output as - -``` -Got version: 0 -Error reading data: version does not exist -``` - -After piping the outputs I used the `diff` command to get the difference between both the shapes. - -``` -# diff shape-1.txt shape-2.txt - -1a2,3 -> *4 00666163746F72792F6A756E6F31686B6B706A6B6D6779676A33777579397473637474616E74766B793239787379747374356D7A2F74657374 -> -3 B2B690FE501397C91C4ED9B17A04F45F480EF71D98F401CDAF10BF792753E3A3 -3c5 -< -3 DF426B01E303332BC40C69CBE92BEBA05E926EB73D2A765857E453B87F2C906A ---- -> -2 36A2B0B9EFB8576635CC592EB8A2767FA1DA8D0D41185E75CCD86BA5EF8EEA9F -5,7c7,9 -< -2 BEADC1C866B478792DC4C5BE3B737438673D5DA3C542B43F9ED01B8A85942FB2 -< *3 02141CA54005142878EC59D5426B352F19524A806A167374616B65 -< -1 F4D4EF086852E6096F07AC20690A6FDB923EBC5E3D18AE849FED4A6EBB571FEC --- -> -3 B746B3645DF5B7FFC0CC0AA1EDA3D6C8CD6A9E206DE5486D3765386045A3BA85 -> *4 02141CA54005142878EC59D5426B352F19524A806A167374616B65 -> -1 89CAD3490194AF4E93AACAEB43B37E227EFFD143C52F56C1EFD7796B2DAB0370 -11,13c13,19 -``` - -From this I could see that there are a few differences in the shapes. - -So I converted the hexadecimal string to ascii to see what it was. -``` -$ echo "00666163746F72792F6A756E6F31686B6B706A6B6D6779676A33777579397473637474616E74766B793239787379747374356D7A2F74657374" | xxd -r -p -factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test -``` +## 1. Setup: Simulating an Apphash Mismatch +- Two-node devnet using [Juno](https://github.com/CosmosContracts/juno/) binaries +- Validator 1: v22.0.0 +- Validator 2: v22.0.1 +- A transaction triggers an apphash mismatch, halting the chain -And that is the culprit of the chain halt. +--- -## How the apphash was triggered +## 2. Step-by-Step Debugging Procedure + +### Step 1: Identify the Offending Module +- Use an apphash calculator (e.g., [apphash_calculator](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23)) on both nodes: + ```sh + go run main.go + ``` +- Example output (Validator 1): + ``` + got commitInfo with 34 stores + index 0: store name 08-wasm, hash e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 + ... + index 3: store name bank, hash 8699243f40720dc1c10f65457f04f7be3e88305e1d571917ab38807dc22d733f + ... + hash: 6a84014dfe03092c897c17b979e47b2374c7df6f0b6a993abee1fee2ee146ce7 + ``` +- Example output (Validator 2): + ``` + got commitInfo with 34 stores + ... + index 3: store name bank, hash eea82414b653f802a9e1127add39e4aac569e72b4b36beb07036a88b3ae726c8 + ... + hash: 94800fc8dc32db79388b5033d40d091b21b870865889a9d0b9e7563db93c44b9 + ``` +- Focus on the module(s) with differing hashes (e.g., `bank`) + +### Step 2: Inspect the Offending Module's IAVL Tree +- Use [iaviewer](https://github.com/cosmos/iavl/tree/master/cmd/iaviewer) to dump the IAVL tree shape for the module: + ```sh + iaviewer shape s/k:/ > shape-.txt + # Example for bank: s/k:bank/ + ``` +- Example output (Validator 1): + ``` + *4 00666163746F72792F6A756E6F31686B6B706A6B6D6779676A33777579397473637474616E74766B793239787379747374356D7A2F74657374 + -3 DF426B01E303332BC40C69CBE92BEBA05E926EB73D2A765857E453B87F2C906A + ... + ``` +- Example output (Validator 2): + ``` + *4 00666163746F72792F6A756E6F31686B6B706A6B6D6779676A33777579397473637474616E74766B793239787379747374356D7A2F74657374 + -3 B2B690FE501397C91C4ED9B17A04F45F480EF71D98F401CDAF10BF792753E3A3 + ... + ``` +- Compare the outputs: + ```sh + diff shape-1.txt shape-2.txt + ``` +- Example diff output: + ``` + 1a2,3 + > *4 00666163746F72792F6A756E6F31686B6B706A6B6D6779676A33777579397473637474616E74766B793239787379747374356D7A2F74657374 + > -3 B2B690FE501397C91C4ED9B17A04F45F480EF71D98F401CDAF10BF792753E3A3 + 3c5 + < -3 DF426B01E303332BC40C69CBE92BEBA05E926EB73D2A765857E453B87F2C906A + --- + > -2 36A2B0B9EFB8576635CC592EB8A2767FA1DA8D0D41185E75CCD86BA5EF8EEA9F + ... + ``` +- Look for differences in the tree structure or keys + +### Step 3: Decode the Difference +- Convert any differing hex keys to ASCII to identify the problematic data: + ```sh + echo "00666163746F72792F6A756E6F31686B6B706A6B6D6779676A33777579397473637474616E74766B793239787379747374356D7A2F74657374" | xxd -r -p + # Output: + factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test + ``` +- The decoded key/value points to the root cause of the mismatch -I first created a new denom on the running devnet using the `tokenfactory` module: +--- -``` -junod tx tokenfactory create-denom test --from juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz --chain-id test -``` +## 3. Example: How the Apphash Mismatch Was Triggered +- Created a new denom using the `tokenfactory` module: + ```sh + junod tx tokenfactory create-denom test --from
--chain-id test + ``` +- Minted tokens for the new denom: + ```sh + junod tx tokenfactory mint 1000000factory/
/test --from
--chain-id test + ``` +- This transaction triggered the mismatch due to a code difference between the two binary versions -Once the new denom was created I tried minting tokens of the new denom using: +--- -``` -junod tx tokenfactory mint 1000000factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test --from juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz --chain-id test -``` +## 4. Tips & Notes +- Module names for iaviewer must be in the format `s/k:/` (e.g., `s/k:staking/`) +- If you see `Error reading data: version does not exist`, check your module name format +- Compare binary versions for recent changes if you find a mismatch -Note: I had created the denom as `test` but the `tokenfactory` module registers it as `factory//`. So to mint tokens from that denom required me to use the enire path in the command. +--- -This `mint` command was the tx that triggered the apphash mismatch between the two validators. The debugging procedure I used to find the root cause of the apphash also points to this. Comparing both the [binary versions](https://github.com/CosmosContracts/juno/compare/v22.0.0...v22.0.1) I can see that the there are changes in the bankactions.go file which caused the apphash mismatch. +## References +- [Juno v22.0.0...v22.0.1 diff](https://github.com/CosmosContracts/juno/compare/v22.0.0...v22.0.1) +- [apphash_calculator gist](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23)