Skip to content

Bittorrent Integration [draft] #1137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 73 commits into
base: master
Choose a base branch
from
Draft

Bittorrent Integration [draft] #1137

wants to merge 73 commits into from

Conversation

marcinczenko
Copy link
Contributor

@marcinczenko marcinczenko commented Feb 28, 2025 •
edited
Loading

Draft of the BitTorrent Codex integration.

Should be enough to see the overall idea in code.

What is there:

  • storeTorrent is equivalent of Codex store - this is the entry point when seeding BitTorrent content into Codex network
  • retrieveTorrent is equivalent of Codex retrieve - entry point when requesting BitTorrent content having only info hash as the input.

To be functional still needs:

  • b-encoding and the actual info-hash validation (now there is only a placeholder around the place where it most probably will happen)
  • streaming pieces and validating them
  • REST API (download)
  • REST API (upload)
  • REST API (fetch torrent manifest only for the given info hash)
  • Try to make the Cid for BitTorrent Manifest a valid Cid - investigation done - not technically feasible and does not add any more security.
  • update docs
  • a bit more testing

See also codex-storage/codex-pm#211.

@marcinczenko marcinczenko self-assigned this Feb 28, 2025
@marcinczenko marcinczenko marked this pull request as draft February 28, 2025 17:11
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 3 times, most recently from bf800ac to 8a608e8 Compare March 5, 2025 14:45
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 1b28c03 to b531696 Compare March 11, 2025 01:05
@marcinczenko marcinczenko mentioned this pull request Mar 13, 2025
Open
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 2 times, most recently from aa9989c to e19065b Compare March 23, 2025 00:44
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 7 times, most recently from 820709c to 4237e8b Compare March 31, 2025 18:34
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 3 times, most recently from 7a76387 to 30414e8 Compare April 23, 2025 16:14
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 7e073ad to 295fabf Compare April 26, 2025 23:49
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch 2 times, most recently from 60f16d1 to 7128f62 Compare May 20, 2025 13:21
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 29c6d93 to 6b88273 Compare June 2, 2025 02:37
marcinczenko added 29 commits June 3, 2025 03:00
@marcinczenko
refactor piece validator tests
ea3cfe0
@marcinczenko
adds torrent downloader abstraction
50ca6da
@marcinczenko
adds testing setup for torrentdownloader
526e3f1
@marcinczenko
do not randomize piece fetching sequence for now
ee2f403
@marcinczenko
adds some good weather tests for torrentdownloader
c859250
@marcinczenko
adds more tests and ability to retrieve the downloaded blocks
2aa91e5
@marcinczenko
updates and integrates torrentdownloader into api and node
534cb71
@marcinczenko
updates copyright
414906a
@marcinczenko
checked exceptions in stores
5abce81
@marcinczenko
better exception handling
b0be205
@marcinczenko
better exception handling in node
edf4627
@marcinczenko
uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"
4d5dfb1
@marcinczenko
uses SafeAsyncIter to stream the blocks in api
6000ae5
@marcinczenko
fix failing test in torrentdownloader
e3fd2fa
@marcinczenko
gets rid of ugly casts
6c32f6b
@marcinczenko
adds more tests to torrentdownloader
f9d2b21
@marcinczenko
gets rid of future casting in torrent downloader
85e0e58
@marcinczenko
Uploading directories - first draft.
912479e
@marcinczenko
moves tarball-related stuff out of node + adds explicit padding argum…
6852c1f 
…ent to "store" + adds "onBatch" callback to "fetchDatasetAsync"
@marcinczenko
Cleans up directory manifest and tartballs
6aad595
@marcinczenko
adds directory downloader
2e6d7d0
@marcinczenko
adds API for retrieving directory
108370e
@marcinczenko
convenience feature - download content using magnet links
8ac8f94
@marcinczenko
Updates magnet link tests
03ff25b
@marcinczenko
adds torrent parser to support native torrent files when downloading
0f62624
@marcinczenko
adds magent link and torrent parser tests to bittorrent tests
f9ce889
@marcinczenko
adds integration tests for using magnet links and torrent file while …
29bb017 
…retrieving BitTorrent content from Codex Net
@marcinczenko
removes not needed err from "without" in repostore test
ff40917
@marcinczenko
checking if auto in the block expiration test has any impact on lib…
0935abb 
…pcre requirement
@marcinczenko marcinczenko force-pushed the feat/bittorrent-integration branch from 74f39bd to 0935abb Compare June 3, 2025 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@marcinczenko marcinczenko

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@marcinczenko