-
Notifications
You must be signed in to change notification settings - Fork 0
Add TPS monitoring tool #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Vsevolod-Rusinskiy
wants to merge
43
commits into
master
Choose a base branch
from
add-tps-monitoring-tool
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
ba0e89c
Add TPS monitoring tool
Vsevolod-Rusinskiy 088f93a
Update .gitignore to include docs/ directory and optimize showStats m…
Vsevolod-Rusinskiy 9b2793d
Refactor: modular TPS monitor with Utils class
Vsevolod-Rusinskiy 883b6bd
Add CSV export to dedicated folder
Vsevolod-Rusinskiy 21c391e
Move common components to shared folder
Vsevolod-Rusinskiy 7175944
Decompose transaction sender into modular architecture
Vsevolod-Rusinskiy 045a33f
Simplify CLI parameters with defaults
Vsevolod-Rusinskiy 138e027
fix: add missing getApi method to ApiConnector
Vsevolod-Rusinskiy 3abc506
Remove csv-report files from tracking and update .gitignore
Vsevolod-Rusinskiy 2670ddb
feat: rename orchestrator to dashboard and update Phase 1 progress
Vsevolod-Rusinskiy c7e9119
feat: implement structured logging system - Phase 3a complete
Vsevolod-Rusinskiy 56eda61
feat: migrate monitor core modules to structured logging
Vsevolod-Rusinskiy 3238559
feat: migrate statistics-reporter to structured logging
Vsevolod-Rusinskiy 43cd860
feat: complete monitor modules logging migration
Vsevolod-Rusinskiy b45cea7
feat: complete sender modules logging migration
Vsevolod-Rusinskiy 0a5ce5f
feat(dashboard): complete TUI component system
Vsevolod-Rusinskiy 263e8a1
Fix TUI dashboard startup and event loop
Vsevolod-Rusinskiy 5598e1d
Fix dashboard layout and API logs display
Vsevolod-Rusinskiy 758beb5
feat: increase log file size limits from 5MB to 20MB
Vsevolod-Rusinskiy 06ba6ee
refactor: remove unused blessed-contrib dependency and simplify TPS g…
Vsevolod-Rusinskiy afe4a34
feat(dashboard): enhance process management and control panel functio…
Vsevolod-Rusinskiy d12b29c
refactor(dashboard): improve keyboard handling and logging
Vsevolod-Rusinskiy c49075b
feat(dashboard): implement file-based logging for console outputs
Vsevolod-Rusinskiy 78f07fe
feat(dashboard): extend file-based logging to include console.warn
Vsevolod-Rusinskiy 4f96afd
refactor(dashboard): update comments for clarity and consistency
Vsevolod-Rusinskiy 6580385
refactor(dashboard): remove general update loop and replace TPS graph…
Vsevolod-Rusinskiy c76c1e7
feat(dashboard): add memory usage logging to EventLogComponent
Vsevolod-Rusinskiy cac9c25
refactor(dashboard): update comments for clarity in logging functiona…
Vsevolod-Rusinskiy 263b4c4
refactor(dashboard): comment out TPS metrics updates due to conflict
Vsevolod-Rusinskiy 1a2fa00
feat(dashboard): implement dynamic log file initialization in LogTPSR…
Vsevolod-Rusinskiy ca178f3
feat(dashboard): enhance logging functionality with custom debug logger
Vsevolod-Rusinskiy 9f6a3e7
refactor(dashboard): simplify structure
Vsevolod-Rusinskiy c9e9ff4
refactor(dashboard): remove unused scripts and files, add documentation
Vsevolod-Rusinskiy e1336eb
chore: update .gitignore to include git directory
Vsevolod-Rusinskiy 93f19b9
refactor(tps-monitoring): update README and package.json
Vsevolod-Rusinskiy 773440d
docs: translate README to English and clean up project structure
Vsevolod-Rusinskiy 416caa2
refactor(tps-monitoring): update author and license in package.json, …
Vsevolod-Rusinskiy 078789e
fix(transaction-sender): add proper nonce management
Vsevolod-Rusinskiy e6008d4
feat(load-testing): add sub-flood load testing tool structure
Vsevolod-Rusinskiy ac44ebe
refactor(tps-monitoring): update package.json and package-lock.json f…
Vsevolod-Rusinskiy b598a2e
refactor(tps-monitoring): update package dependencies and clean up pa…
Vsevolod-Rusinskiy c11de81
chore(tps-monitoring): update .gitignore and tsconfig.json, enhance p…
Vsevolod-Rusinskiy 93ee192
refactor(tps-monitoring): add detailed comments
Vsevolod-Rusinskiy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,3 +6,4 @@ | |
/.idea | ||
.env | ||
.DS_Store | ||
git/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
node_modules/ | ||
|
||
|
||
|
||
*.log | ||
.DS_Store | ||
|
||
# Drafts folder for temporary files and notes | ||
drafts/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,379 @@ | ||
# 🚀 Simple TPS Monitor: Substrate TPS Measurement Tool | ||
|
||
A simple tool for measuring real TPS (transactions per second) in Polkadot/Substrate blockchains with modular architecture. | ||
|
||
## 🎯 Features | ||
|
||
- ✅ **Accurate TPS measurement** - analyzes blocks, not sending speed | ||
- ✅ **Real balance transactions** - uses transferKeepAlive | ||
- ✅ **Automatic nonce management** - proper handling of multiple transactions | ||
- ✅ **Continuous monitoring** - tracks blocks in real-time | ||
- ✅ **Configurable load** - --tps parameter for controlling sending frequency | ||
- ✅ **Monitor-only mode** - --tps 0 for analysis without load generation | ||
- ✅ **Simple CLI interface** - one command to run | ||
- ✅ **Graceful shutdown** - proper termination with Ctrl+C | ||
- ✅ **Modular architecture** - clean separation of concerns | ||
|
||
## 📦 Installation | ||
|
||
```bash | ||
# Clone the repository | ||
git clone https://github.com/your-username/tps-monitoring.git | ||
cd tps-monitoring | ||
|
||
# Install dependencies | ||
npm install | ||
``` | ||
|
||
## 🔧 Requirements | ||
|
||
- Node.js >= 16.0.0 | ||
- Access to Polkadot/Substrate node via WebSocket | ||
- Alice account with balance for testing | ||
|
||
## 🏗️ How It Works - Modular Architecture | ||
|
||
### Project Structure | ||
``` | ||
tps-monitoring/ | ||
├── src/ | ||
│ ├── index.js # CLI entry point | ||
│ ├── modules/ | ||
│ │ ├── TPSMonitor.js # Main coordinator | ||
│ │ ├── TransactionSender.js # Transaction generation | ||
│ │ └── BlockMonitor.js # Block monitoring & TPS calculation | ||
│ └── simple-monitor.js # Legacy monolithic version | ||
``` | ||
|
||
### System Flow | ||
|
||
#### 1. **Entry Point (index.js)** | ||
```javascript | ||
// Parse CLI arguments and create TPSMonitor instance | ||
const monitor = new TPSMonitor() | ||
await monitor.start(options) | ||
``` | ||
|
||
**What happens:** | ||
- Parses command line arguments (`--node`, `--tps`) | ||
- Creates TPSMonitor instance | ||
- Starts the monitoring system | ||
|
||
#### 2. **Main Coordinator (TPSMonitor.js)** | ||
```javascript | ||
// Initialize blockchain connection and components | ||
await this.initialize(options.node) | ||
this.blockMonitor.startMonitoring(this.api) | ||
if (options.tps > 0) { | ||
await this.transactionSender.startSending(options.tps) | ||
} | ||
``` | ||
|
||
**What happens:** | ||
- Connects to blockchain via WebSocket | ||
- Initializes cryptography and keyring | ||
- Creates TransactionSender and BlockMonitor instances | ||
- Starts both monitoring and transaction sending (if enabled) | ||
- Handles graceful shutdown on Ctrl+C | ||
|
||
#### 3. **Transaction Sender (TransactionSender.js)** | ||
```javascript | ||
// Send transactions at specified TPS rate | ||
const intervalMs = 1000 / tpsTarget | ||
const sendTx = async () => { | ||
const transfer = this.api.tx.balances.transferKeepAlive(this.alice.address, 1) | ||
await transfer.signAndSend(this.alice) | ||
setTimeout(sendTx, intervalMs) | ||
} | ||
``` | ||
|
||
**What happens:** | ||
- Creates Alice test account | ||
- Sends `transferKeepAlive` transactions to self | ||
- Maintains specified TPS rate using `setTimeout` | ||
- Logs progress every 10 transactions | ||
- Can be stopped gracefully | ||
|
||
#### 4. **Block Monitor (BlockMonitor.js)** | ||
```javascript | ||
// Subscribe to new blocks and calculate TPS | ||
api.derive.chain.subscribeNewHeads(async (header) => { | ||
const block = await api.rpc.chain.getBlock(header.hash) | ||
const txCount = block.block.extrinsics.length | ||
|
||
// Store data and calculate TPS | ||
this.blockTimes.push(now) | ||
this.txCounts.push(txCount) | ||
const avgTPS = this.calculateTPS() | ||
}) | ||
``` | ||
|
||
**What happens:** | ||
- Subscribes to new blocks via WebSocket | ||
- Extracts transaction count from each block | ||
- Maintains sliding window of last 10 blocks | ||
- Calculates real-time TPS: `total_transactions / time_span` | ||
- Displays statistics for each block | ||
|
||
### Component Interaction | ||
|
||
``` | ||
┌─────────────┐ ┌─────────────────┐ ┌─────────────────┐ | ||
│ index.js │───▶│ TPSMonitor.js │───▶│TransactionSender│ | ||
│ (CLI) │ │ (Coordinator) │ │ .js │ | ||
└─────────────┘ └─────────────────┘ └─────────────────┘ | ||
│ │ | ||
▼ │ | ||
┌─────────────────┐ │ | ||
│ BlockMonitor.js │◀────────────┘ | ||
│ (TPS Calculator)│ (API shared) | ||
└─────────────────┘ | ||
``` | ||
|
||
### Key Benefits of Modular Design | ||
|
||
#### **Separation of Concerns** | ||
- **TransactionSender**: Only handles transaction generation | ||
- **BlockMonitor**: Only handles block monitoring and TPS calculation | ||
- **TPSMonitor**: Only handles coordination and lifecycle | ||
- **index.js**: Only handles CLI interface | ||
|
||
#### **Testability** | ||
```javascript | ||
// Test components independently | ||
const sender = new TransactionSender(api, keyring) | ||
await sender.startSending(5) | ||
|
||
const monitor = new BlockMonitor() | ||
monitor.startMonitoring(api) | ||
``` | ||
|
||
#### **Reusability** | ||
- TransactionSender can be used in other load testing tools | ||
- BlockMonitor can be extended for other blockchain metrics | ||
- TPSMonitor can be adapted for different blockchain networks | ||
|
||
#### **Maintainability** | ||
- Easy to find and fix bugs in specific components | ||
- Simple to add new features (e.g., different transaction types) | ||
- Clear interfaces between components | ||
|
||
## 🚀 Usage | ||
|
||
### Basic Command | ||
|
||
```bash | ||
# Run with load generation (10 TPS by default) | ||
node src/index.js | ||
|
||
# Connect to custom node | ||
node src/index.js --node ws://your-node:9944 | ||
|
||
# Set target TPS | ||
node src/index.js --tps 50 | ||
|
||
# Monitor only without load generation | ||
node src/index.js --tps 0 | ||
|
||
# Full example | ||
node src/index.js \ | ||
--node ws://localhost:9944 \ | ||
--tps 25 | ||
``` | ||
|
||
### CLI Parameters | ||
|
||
- `-n, --node <url>` - Node WebSocket URL (default: ws://localhost:9944) | ||
- `-t, --tps <number>` - Target TPS to generate (0 = monitor only, default: 10) | ||
|
||
### NPM Scripts | ||
|
||
```bash | ||
# Run with default parameters | ||
npm start | ||
``` | ||
|
||
## 📊 How it works | ||
|
||
### 1. Blockchain Connection | ||
- Initializes cryptography | ||
- Connects to node via WebSocket | ||
- Creates Alice account for testing | ||
|
||
### 2. Load Generation (if --tps > 0) | ||
- Sends Alice → Alice transactions at specified frequency | ||
- Uses `transferKeepAlive` for safe transfers | ||
- Automatically manages nonce for each transaction | ||
- Shows progress every 10 transactions | ||
|
||
### 3. TPS Monitoring | ||
- Subscribes to new blocks | ||
- Counts transactions in each block | ||
- Calculates average TPS over last 10 blocks | ||
- Shows statistics in real-time | ||
|
||
### 4. Statistics Output | ||
``` | ||
Block #1234: 8 txs, 13.3 TPS (45s runtime) | ||
``` | ||
Where: | ||
- `#1234` - block number | ||
- `8 txs` - number of transactions in block | ||
- `13.3 TPS` - average TPS over last 10 blocks | ||
- `45s runtime` - program runtime | ||
|
||
## 🏃♂️ Usage Examples | ||
|
||
### Node Performance Testing | ||
|
||
```bash | ||
# Start with low load | ||
node src/index.js --tps 5 | ||
|
||
# Gradually increase load | ||
node src/index.js --tps 10 | ||
node src/index.js --tps 20 | ||
node src/index.js --tps 50 | ||
``` | ||
|
||
### Monitoring Existing Network | ||
|
||
```bash | ||
# Monitor TPS without generating load | ||
node src/index.js --tps 0 --node ws://mainnet-node:9944 | ||
``` | ||
|
||
### Local Testing | ||
|
||
```bash | ||
# Test local node | ||
node src/index.js --node ws://localhost:9944 --tps 15 | ||
``` | ||
|
||
## 🔍 Differences from Legacy Script | ||
|
||
### ❌ Problems in original script: | ||
1. **Wrong transaction type**: `system.remark` instead of `balances.transfer` | ||
2. **Automatic nonce**: led to transaction rejections | ||
3. **Sending speed measurement**: instead of real TPS | ||
4. **One-time execution**: send batch and exit | ||
|
||
### ✅ Fixes in Simple TPS Monitor: | ||
1. **Balance transfers** - real money transfers | ||
2. **Automatic nonce** - proper handling of multiple transactions | ||
3. **Block reading** - real TPS measurement | ||
4. **Continuous operation** - configurable sending frequency | ||
5. **Block monitoring** - real-time analysis | ||
6. **Modular architecture** - clean separation of concerns | ||
|
||
## 🛠️ Troubleshooting | ||
|
||
### Transactions not going through: | ||
- Check Alice account balance | ||
- Ensure node is accessible and running | ||
- Verify node URL is correct | ||
|
||
### Low TPS: | ||
- Try reducing sending frequency (--tps) | ||
- Check node load | ||
- Ensure node doesn't limit TPS | ||
|
||
### Connection errors: | ||
- Check node accessibility | ||
- Ensure correct WebSocket URL is used | ||
- Check network settings | ||
|
||
### Common Error Messages: | ||
- `Priority is too low`: Transaction pool is full, reduce TPS | ||
- `Transaction is outdated`: Nonce issue, restart the tool | ||
- `Connection failed`: Check node URL and network | ||
|
||
## 📈 Load Testing Recommendations | ||
|
||
```bash | ||
# 1. Start with monitoring without load | ||
node src/index.js --tps 0 | ||
|
||
# 2. Add minimal load | ||
node src/index.js --tps 1 | ||
|
||
# 3. Gradually increase load | ||
node src/index.js --tps 5 | ||
node src/index.js --tps 10 | ||
node src/index.js --tps 20 | ||
|
||
# 4. Find maximum load | ||
node src/index.js --tps 50 | ||
node src/index.js --tps 100 | ||
``` | ||
|
||
## 🎯 Result | ||
|
||
Simple TPS Monitor provides: | ||
- ✅ **Accurate TPS measurement** - based on blockchain data | ||
- ✅ **Proper transaction sending** - with nonce increment | ||
- ✅ **Real load** - balance transfers, not just messages | ||
- ✅ **Configuration flexibility** - --tps parameter for load control | ||
- ✅ **Ease of use** - single file, one command launch | ||
- ✅ **Modular design** - clean architecture, easy to extend | ||
|
||
**Result:** Reliable and accurate tool for measuring Polkadot/Substrate network performance. | ||
|
||
## 📋 Requirements for Testing | ||
|
||
### Node Setup | ||
- Running Substrate/Polkadot node | ||
- WebSocket endpoint available (default: ws://localhost:9944) | ||
- Node configured for development/testing | ||
|
||
### Account Setup | ||
- Alice account with sufficient balance | ||
- Account should be able to send transactions | ||
- Recommended: Use development node with pre-funded accounts | ||
|
||
## 🔧 Development | ||
|
||
### Dependencies | ||
- `@polkadot/api` - Polkadot/Substrate API | ||
- `@polkadot/util-crypto` - Cryptographic utilities | ||
- `commander` - CLI argument parsing | ||
|
||
### Architecture Benefits | ||
|
||
#### **Single Responsibility Principle (SRP)** | ||
- Each class has one responsibility | ||
- Easier to understand and maintain | ||
|
||
#### **Testability** | ||
```javascript | ||
// Can test components separately | ||
const sender = new TransactionSender(api, keyring) | ||
await sender.startSending(5) | ||
|
||
const monitor = new BlockMonitor() | ||
monitor.startMonitoring(api) | ||
``` | ||
|
||
#### **Reusability** | ||
- TransactionSender can be used in other projects | ||
- BlockMonitor can be extended for other metrics | ||
|
||
#### **Extensibility** | ||
- Easy to add new transaction types | ||
- Can add new metrics to BlockMonitor | ||
|
||
## 📄 License | ||
|
||
This project is unlicensed. All rights reserved. | ||
|
||
## 🤝 Contributing | ||
|
||
1. Fork the repository | ||
2. Create your feature branch | ||
3. Commit your changes | ||
4. Push to the branch | ||
5. Create a Pull Request | ||
|
||
--- | ||
|
||
**Note:** This tool is designed for testing and development purposes. Use responsibly and ensure you have proper permissions for the target network. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add
.nvmrc
.