Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 37 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,62 +1,53 @@
# Apphashd

This is a framework for identifying the root cause of an appphash mismatch on cosmos based networks inspired by [apphash_calculator](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23) written by [freak12techno](https://github.com/freak12techno).

The `apphashd` binary takes two inputs as arguments, absolute/relative paths to the `application.db` of nodes with opposing apphashes. It calculates and compares the apphashes of all the modules present in the db. It generates an array with the name of the modules which have differing hashes.

This repo contains a script called `apphashd.sh` which is an end to end script which builds the `apphashd` binary, executes it and uses [iaviewer](https://github.com/cosmos/iavl/tree/master/cmd/iaviewer) to identify the message_type/data that caused the apphash to deviate on both the nodes.

## Usage
Download and run the script
```
wget https://raw.githubusercontent.com/vitwit/apphashd/main/apphashd.sh && chmod 755 apphashd.sh
./apphashd.sh <path-to-application.db-node1> <path-to-application.db-node2>
```
The script creates a folder called `hashes` inside the `apphashd` folder which contains the module hashes of both the nodes, iavl trees of the differing modules, diff of iavl trees and the ascii decoded output of the diff which contains the root cause of apphash mismatch.

### Example usage
This repo has a folder called `testdata` which contains the application.db of two nodes which were halted due to an apphash mismatch. We will use that to test the script.

#### Download the testdata
```
A tool to quickly identify the root cause of apphash mismatches on Cosmos-based networks. Inspired by [apphash_calculator](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23).

## Features
- Compares two `application.db` directories from different nodes
- Calculates and compares module apphashes
- Identifies modules with differing hashes
- Uses [iaviewer](https://github.com/cosmos/iavl/tree/master/cmd/iaviewer) to pinpoint the exact data causing the mismatch

## Quick Start
1. **Download and run the script:**
```sh
wget https://raw.githubusercontent.com/vitwit/apphashd/main/apphashd.sh && chmod 755 apphashd.sh
./apphashd.sh <path-to-application.db-node1> <path-to-application.db-node2>
```
2. **Results:**
- A `hashes` folder is created inside `apphashd`.
- Contains module hashes, IAVL trees, diffs, and decoded output showing the root cause of the mismatch.

## Usage Example
This repo includes test data from two nodes with an apphash mismatch.

### 1. Download test data
Copy link

Copilot AI Jun 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Using separate header levels for each step under 'Usage Example' can be verbose; consider using a nested numbered list to match the 'Quick Start' format and improve readability.

Copilot uses AI. Check for mistakes.
```sh
mkdir ~/node1 ~/node2
cd ~/node1
wget https://github.com/vitwit/apphashd/raw/main/testdata/node1/node1-application-db.tar.gz
tar -xvf node1-application-db.tar.gz
tar -xvf node1-application-db.tar.gz
cd ~/node2
wget https://github.com/vitwit/apphashd/raw/main/testdata/node2/node2-application-db.tar.gz
tar -xvf node2-application-db.tar.gz
```

#### Download and execute the script
```
### 2. Run the script
```sh
cd ~/
wget https://raw.githubusercontent.com/vitwit/apphashd/main/apphashd.sh && chmod 755 apphashd.sh
./apphashd.sh ~/node1/application.db ~/node2/application.db
```

The script generated a file called `diff-bank-decoded` inside the `~/apphashd/hashes` folder. We can check the root cause of the apphash mismatch by inspecting this file.
```
factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test
Ebzf>�����`Up6ɳk\vW�]Z��;Y
��6��N�.�v�X7m�;��&�\�ӻ�6Q
�X?R�%�����J��ğ5��'�І�qc�xv
�!9�E�ErB�l&��E��d^;��2���
�@(x�Y�Bk5/RJ�jstake
woGϬ�U��ds*�;�+e�,ʣ^�
Q@�Sn��������n���)�H���3�
�@(x�Y�Bk5/RJ�jstake
��Mۗ�幗�t��pW�cU�N,��A�=jG�O
?,vD�t�S���Ѽ/lBq�+��cR��>n
��[h"%p�\0��ke���factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test
s;�ܝ"s�]�����<u�+"��rs{n�
��[h"%p�\0��ke���stake
�D����f�V/�<4@����$�6?r�I���
factory/juno1hkkpjkmgygj3wuy9tscttantvky29xsytst5mz/test��[h"%p�\0��ke���
���D��?",
���я9/�4٥ !����KU�/[}�_
��[h"%p�\0��ke���stake
����v���������Q���*���)b��
```
### 3. Inspect the results
- Check the `diff-bank-decoded` file in `~/apphashd/hashes` for the root cause of the mismatch.

## How it Works
- Compares module hashes between two nodes
- Finds modules with differences
- Uses iaviewer to extract and diff IAVL trees
- Decodes the diff to reveal the exact data discrepancy

Refer to this [doc](./procedure.md) to know how the apphash mismatch was simulated in a two node devnet and how [freak12techno's](https://github.com/freak12techno) [apphash_calculator](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23) was used to debug it originally which served as the inspiration for `apphashd`
## More Information
- See [procedure.md](./procedure.md) for details on how the mismatch was simulated and debugged.
- Inspired by [freak12techno](https://github.com/freak12techno)'s [apphash_calculator](https://gist.github.com/freak12techno/845a3061ed65295667c145c05ffd3b23).
Loading