-
Notifications
You must be signed in to change notification settings - Fork 24
feat: add web3 services and storage #1580
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Kaladin13
wants to merge
11
commits into
main
Choose a base branch
from
ton-storage
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+401
−15
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
d037b6d
add web3 services and storage
Kaladin13 a6435ca
fixes
Kaladin13 cd96adb
redirects
Kaladin13 cee144f
more redi
Kaladin13 1dba157
add to start here table
Kaladin13 0dba3dd
add torrent and merkle
Kaladin13 3b11731
changes
Kaladin13 5f779a0
Merge branch 'main' into ton-storage
Gusarich ea9a11d
fix review
Kaladin13 597a8ba
Merge remote-tracking branch 'origin/ton-storage' into ton-storage
Kaladin13 883d210
Merge branch 'main' into ton-storage
Kaladin13 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| --- | ||
| title: "Run storage daemon" | ||
| sidebarTitle: "Storage daemon" | ||
| --- | ||
|
|
||
| import { Aside } from "/snippets/aside.jsx"; | ||
|
|
||
| `storage-daemon` downloads and shares TON Storage bags. `storage-daemon-cli` is the console client used to control it. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Linux, macOS, or Windows host with at least a dual-core 1 GHz CPU, 2 GB RAM, 2 GB SSD (plus space for bags), and stable 10 Mb/s bandwidth with a public/static IP | ||
| - Global network config file, for example `global.config.json` from the [TON configs](https://ton-blockchain.github.io/) | ||
|
|
||
| ## Install storage-daemon | ||
|
|
||
| - Download prebuilt `storage-daemon` and `storage-daemon-cli` binaries from [TON auto builds](https://github.com/ton-blockchain/ton/releases/latest) (built from the `testnet` branch). | ||
| - Or build from source in the [`ton-blockchain/ton`](https://github.com/ton-blockchain/ton/tree/testnet) repository (target: `storage-daemon` and `storage-daemon-cli`). | ||
|
|
||
| ## Start the daemon | ||
|
|
||
| Run the daemon with explicit ports and a data directory: | ||
|
|
||
| ```bash | ||
| storage-daemon -v 3 -C global.config.json -I <IP>:3333 -p 5555 -D storage-db | ||
| ``` | ||
|
|
||
| - `-v` — log verbosity (`3` = INFO) | ||
| - `-C` — path to the global network config | ||
| - `-I` — ADNL listen IP and port (`<IP>` is the public or reachable address of the host) | ||
| - `-p` — TCP port for the console interface | ||
| - `-D` — path to the daemon database; CLI keys are generated inside the `cli-keys/` subdirectory on first start | ||
|
|
||
| <Aside type="tip"> | ||
| Open the console port in the firewall only when needed. Keep the daemon bound to `localhost` when `storage-daemon-cli` runs on the same host. | ||
| </Aside> | ||
|
|
||
| ## Connect with storage-daemon-cli | ||
|
|
||
| Point the CLI at the console port and keypair generated by the daemon: | ||
|
|
||
| ```bash | ||
| storage-daemon-cli -I 127.0.0.1:5555 -k storage-db/cli-keys/client -p storage-db/cli-keys/server.pub | ||
| ``` | ||
|
|
||
| - `-I` — daemon host and console port | ||
| - `-k` — client private key path | ||
| - `-p` — server public key path (CLI option, not a port) | ||
|
|
||
| `storage-daemon-cli` can also run non-interactively: | ||
|
|
||
| ```bash | ||
| storage-daemon-cli ... -c "add-by-meta bag-meta.bag" -c "list --hashes" | ||
| ``` | ||
|
|
||
| ## Add bags for download | ||
|
|
||
| Download by bag ID hash or by meta file: | ||
|
|
||
| ```bash | ||
| add-by-hash <BAG_ID> -d <DOWNLOAD_DIR> | ||
| add-by-meta <META_FILE> -d <DOWNLOAD_DIR> | ||
| ``` | ||
|
|
||
| - `<BAG_ID>` — 64-character hex bag ID | ||
| - `<DOWNLOAD_DIR>` — target directory for downloaded files | ||
| - `<META_FILE>` — path to the bag meta file | ||
| - If `-d` is omitted, files are saved to the daemon's default storage directory. | ||
|
|
||
| `list` shows added bags; `list --hashes` prints full IDs. Use either the numeric ordinal from `list` (valid only for the current session) or the full ID in other commands. | ||
|
|
||
| ### Inspect and control transfers | ||
|
|
||
| - `get <BAG_ID>` — show description, size, speed, and file list | ||
| - `get-peers <BAG_ID>` — list connected peers | ||
| - `download-pause <BAG_ID>` / `download-resume <BAG_ID>` — control downloads | ||
| - `upload-pause <BAG_ID>` / `upload-resume <BAG_ID>` — control uploads | ||
| - `remove <BAG_ID>` — remove the bag | ||
| - `remove --remove-files <BAG_ID>` — remove the bag and files (files are always removed if stored in the daemon's internal directory) | ||
|
|
||
| <Aside | ||
| type="caution" | ||
| > | ||
| `remove --remove-files` deletes local files. Run it only when the files are backed up or no longer needed. | ||
| </Aside> | ||
|
|
||
| ## Partial downloads and priorities | ||
|
|
||
| Download only selected files by setting priorities: | ||
|
|
||
| ```bash | ||
| add-by-hash <BAG_ID> -d <DOWNLOAD_DIR> --partial file1 file2 | ||
| add-by-meta <META_FILE> -d <DOWNLOAD_DIR> --partial file1 file2 | ||
| ``` | ||
|
|
||
| Each file has a priority `0–255`; `0` disables download. `--partial` sets listed files to `1` and all others to `0`. | ||
|
|
||
| Update priorities after adding the bag: | ||
|
|
||
| - `priority-all <BAG_ID> <priority>` — set the same priority for every file | ||
| - `priority-idx <BAG_ID> <index> <priority>` — set by file index from `get` | ||
| - `priority-name <BAG_ID> <name> <priority>` — set by file name | ||
|
|
||
| Priorities can be set before the full file list loads. | ||
|
|
||
| - `<priority>` — integer priority from `0` (do not download) to `255` (highest) | ||
| - `<index>` — file index from the `get` output | ||
| - `<name>` — file name from the `get` output | ||
|
|
||
| ## Create and share a bag | ||
|
|
||
| Create a bag from a file or directory and start seeding: | ||
|
|
||
| ```bash | ||
| create <PATH> -d "Description of the bag" | ||
| ``` | ||
|
|
||
| - `<PATH>` — file or directory to include in the bag | ||
|
|
||
| Options: | ||
|
|
||
| - `--no-upload` — create the bag without uploading; enable later with `upload-resume` | ||
| - `--copy` — copy files into the daemon's internal directory before seeding | ||
|
|
||
| After creation the CLI prints the bag ID. Share the meta file if others need a `.torrent`-like descriptor: | ||
|
|
||
| ```bash | ||
| get-meta <BAG_ID> <META_FILE> | ||
| ``` | ||
|
|
||
| - `<META_FILE>` — path to save the generated meta file |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| --- | ||
| title: "TON Storage FAQ" | ||
| sidebarTitle: "FAQ" | ||
| --- | ||
|
|
||
| ## Assign a TON domain to a TON Storage bag | ||
|
|
||
| 1. [Create and upload a bag](/services/storage/daemon#create-and-share-a-bag) and copy its bag ID `<BAG_ID>` (64-character hex). | ||
| 1. Open Google Chrome. | ||
| 1. Install a wallet browser extension: | ||
| - [TON Wallet](https://chrome.google.com/webstore/detail/ton-wallet/nphplpgoakhhjchkkhmiggakijnkhfnd) | ||
| - [MyTonWallet](https://chrome.google.com/webstore/detail/mytonwallet/fldfpgipfncgndfolcbkdeeknbbbnhcc) | ||
| 1. Import the wallet that owns the domain using the recovery phrase. | ||
| 1. Go to [dns.ton.org](https://dns.ton.org), open the domain, and select **Edit**. | ||
| 1. Paste `<BAG_ID>` into the **Storage** field and save. | ||
|
|
||
| ## Host a static TON Site from TON Storage | ||
|
|
||
| 1. Place the static site files in a folder that includes `index.html`. | ||
| 1. [Create a bag](/services/storage/daemon#create-and-share-a-bag) from the folder and upload it. Copy the bag ID. | ||
| 1. Install TON Wallet or MyTonWallet in Chrome and import the domain owner's wallet. | ||
| 1. Go to [dns.ton.org](https://dns.ton.org), open the domain, and select **Edit**. | ||
| 1. Paste `<BAG_ID>` into the **Site** field, choose **Host in TON Storage**, and save. | ||
|
|
||
| ## Migrate NFT content to TON Storage | ||
|
|
||
| For the [standard NFT collection contract](https://github.com/ton-blockchain/token-contract/blob/main/nft/nft-collection-editable.fc), update the content prefix by sending a message from the collection owner's wallet to the collection contract. | ||
|
|
||
| - Old prefix example: `https://mysite/my_collection/` | ||
| - New prefix format: `tonstorage://<BAG_ID>/` | ||
|
|
||
| After updating the prefix, upload refreshed metadata to TON Storage using the new bag ID. | ||
|
|
||
| ## Low-level DNS records for TON Storage | ||
|
|
||
| Assign a bag to the DNS record with key `sha256("storage")`: | ||
|
|
||
| ```tlb | ||
| dns_storage_address#7473 bag_id:uint256 = DNSRecord; | ||
| ``` | ||
|
|
||
| Assign a static site bag directly to the DNS record with key `sha256("site")`: | ||
|
|
||
| ```tlb | ||
| dns_storage_address#7473 bag_id:uint256 = DNSRecord; | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| --- | ||
| title: "TON Storage overview" | ||
| sidebarTitle: "Overview" | ||
| --- | ||
|
|
||
| import { Aside } from "/snippets/aside.jsx"; | ||
|
|
||
| TON Storage is TON's torrent-like distributed file storage layer. It lets services and users publish **bags of files** and retrieve them over [ADNL and RLDP](/foundations/whitepapers/ton#3-1-abstract-datagram-network-layer) with verifiable [Merkle proofs](/foundations/serialization/merkle). Bags stay off-chain; Merkle proofs let storage flows pair with smart contracts, such as storage providers, when enforceable on-chain terms are needed. | ||
|
|
||
| ## What TON Storage delivers | ||
|
|
||
| - Distribute large files without placing data on-chain | ||
| - Prove that a storage provider still holds a bag through on-chain proofs | ||
| - Attach bags to TON DNS domains for human-readable access and TON Sites | ||
| - Run private or public providers that charge for storage | ||
|
|
||
| ## Core pieces | ||
|
|
||
| - **Bag of files**: a torrent-like package. Its Merkle root (the **bag ID**) uniquely identifies the bag. The meta file (`.bag` equivalent) contains torrent info and header data. | ||
| - **storage-daemon** and **storage-daemon-cli**: download, seed, and manage bags. The CLI controls a running daemon. | ||
| - **Storage provider contracts**: smart contracts that accept paid storage requests, manage balances, and verify proofs. | ||
| - **Networking**: data travels over ADNL using RLDP. Each bag uses its own overlay network, and peers are discovered through TON DHT. | ||
|
|
||
| <Aside type="note"> | ||
| TON Storage reuses torrent-style terms (torrent, bag, pieces) but transports data over ADNL/RLDP instead of BitTorrent. | ||
| </Aside> | ||
|
|
||
| ## When to use TON Storage | ||
|
|
||
| - Host static TON Sites directly from TON Storage | ||
| - Distribute application assets, media, or snapshots with verifiable integrity | ||
| - Operate a paid storage service with proof-of-storage guarantees | ||
|
|
||
| ## Next steps | ||
|
|
||
| - Run the daemon and work with bags: [Storage daemon](/services/storage/daemon) | ||
| - Operate or consume paid storage: [Storage provider](/services/storage/provider) | ||
| - Common flows and DNS records: [TON Storage FAQ](/services/storage/faq) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.