Skip to content

Add gearhash, xetchunk and blake3 WASM packages #1500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
814380f
init gearhash package
coyotte508 May 28, 2025
68b32be
fix build command
coyotte508 May 28, 2025
14a9ef4
Create gearhash function from rust source
coyotte508 May 30, 2025
a2cb917
switch to gearhash-wasm package
coyotte508 May 30, 2025
97f3e0b
v 1.0
coyotte508 May 30, 2025
512801f
remove extra file
coyotte508 May 30, 2025
07a384d
bigger mask
coyotte508 May 30, 2025
3306f2c
add nextMatches function
coyotte508 May 30, 2025
12f9e97
(wip) xet chunk code generated by cursor to fix
coyotte508 Jun 4, 2025
36348fe
add blake3-wasm and xetchunk-wasm
coyotte508 Jun 4, 2025
07b115d
fix TS refs
coyotte508 Jun 4, 2025
2664cba
use builtins
coyotte508 Jun 4, 2025
031dc0d
package org
coyotte508 Jun 4, 2025
01d7472
also provide direct wasm exports
coyotte508 Jun 4, 2025
2b9b57a
proper build outputs
coyotte508 Jun 4, 2025
af2ccc1
add exports for JS
coyotte508 Jun 4, 2025
f1a31cb
add test for blake3 + simple function
coyotte508 Jun 13, 2025
fef8423
add rust reference implementation in package
coyotte508 Jun 13, 2025
48996e1
fix mixing function
coyotte508 Jun 13, 2025
eff0eeb
Blake 3 workspnpm --filter blake3-wasm build && pnpm --filter blake3-…
coyotte508 Jun 13, 2025
d87e9b0
improvements
coyotte508 Jun 18, 2025
f47f07d
only output length
coyotte508 Jun 18, 2025
aa44db2
add vendor test for gearhash
coyotte508 Jun 18, 2025
e3d8cdc
comment on mask param
coyotte508 Jun 18, 2025
13c607a
wip try to match rust & assembly
coyotte508 Jun 18, 2025
ea05ede
fix gearhash test
coyotte508 Jun 18, 2025
4270e20
Merge remote-tracking branch 'origin/main' into gearhash-experiment
coyotte508 Jul 7, 2025
fb8e5aa
add splitmix64-wasm module
coyotte508 Jul 7, 2025
3ec13eb
const data test passes
coyotte508 Jul 7, 2025
265c26e
fully test random data
coyotte508 Jul 7, 2025
275eb60
fix mask in wasm xet chunking
coyotte508 Jul 7, 2025
dd1bd54
fix test
coyotte508 Jul 8, 2025
3be827a
change gearhash tests to wasm
coyotte508 Jul 8, 2025
c4f4053
small opt
coyotte508 Jul 8, 2025
3228194
publish wasm packages
coyotte508 Jul 8, 2025
6386e73
add test for chunk hashes
coyotte508 Jul 8, 2025
a02c55c
support for blake3 keyed
coyotte508 Jul 8, 2025
d6a8609
add tests with xet data key
coyotte508 Jul 8, 2025
d8f4348
fix hash => datahash conversion
coyotte508 Jul 8, 2025
1fa349e
add xorb hash, my bad, thought it was already done
coyotte508 Jul 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions .github/workflows/blake3-wasm-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Blake3 WASM - Version and Release

on:
workflow_dispatch:
inputs:
newversion:
type: choice
description: "Semantic Version Bump Type"
default: patch
options:
- patch
- minor
- major

concurrency:
group: "push-to-main"

defaults:
run:
working-directory: packages/blake3-wasm

jobs:
version_and_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
# Needed to push the tag and the commit on the main branch, otherwise we get:
# > Run git push --follow-tags
# remote: error: GH006: Protected branch update failed for refs/heads/main.
# remote: error: Changes must be made through a pull request. Required status check "lint" is expected.
token: ${{ secrets.BOT_ACCESS_TOKEN }}
- run: npm install -g corepack@latest && corepack enable
- uses: actions/setup-node@v3
with:
node-version: "20"
cache: "pnpm"
cache-dependency-path: |
packages/blake3-wasm/pnpm-lock.yaml
# setting a registry enables the NODE_AUTH_TOKEN env variable where we can set an npm token. REQUIRED
registry-url: "https://registry.npmjs.org"
- run: pnpm install
- run: git config --global user.name machineuser
- run: git config --global user.email [email protected]
- run: |
PACKAGE_VERSION=$(node -p "require('./package.json').version")
BUMPED_VERSION=$(node -p "require('semver').inc('$PACKAGE_VERSION', '${{ github.event.inputs.newversion }}')")
# Update package.json with the new version
node -e "const fs = require('fs'); const package = JSON.parse(fs.readFileSync('./package.json')); package.version = '$BUMPED_VERSION'; fs.writeFileSync('./package.json', JSON.stringify(package, null, '\t') + '\n');"
git commit . -m "🔖 @huggingface/blake3-wasm $BUMPED_VERSION"
git tag "blake3-wasm-v$BUMPED_VERSION"
- run: pnpm publish --no-git-checks .
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- run: (git pull --rebase && git push --follow-tags) || (git pull --rebase && git push --follow-tags)
# hack - reuse actions/setup-node@v3 just to set a new registry
- uses: actions/setup-node@v3
with:
node-version: "20"
registry-url: "https://npm.pkg.github.com"
# Disable for now, until github supports PATs for writing github packages (https://github.com/github/roadmap/issues/558)
# - run: pnpm publish --no-git-checks .
# env:
# NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
64 changes: 64 additions & 0 deletions .github/workflows/gearhash-wasm-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Gearhash WASM - Version and Release

on:
workflow_dispatch:
inputs:
newversion:
type: choice
description: "Semantic Version Bump Type"
default: patch
options:
- patch
- minor
- major

concurrency:
group: "push-to-main"

defaults:
run:
working-directory: packages/gearhash-wasm

jobs:
version_and_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
# Needed to push the tag and the commit on the main branch, otherwise we get:
# > Run git push --follow-tags
# remote: error: GH006: Protected branch update failed for refs/heads/main.
# remote: error: Changes must be made through a pull request. Required status check "lint" is expected.
token: ${{ secrets.BOT_ACCESS_TOKEN }}
- run: npm install -g corepack@latest && corepack enable
- uses: actions/setup-node@v3
with:
node-version: "20"
cache: "pnpm"
cache-dependency-path: |
packages/gearhash-wasm/pnpm-lock.yaml
# setting a registry enables the NODE_AUTH_TOKEN env variable where we can set an npm token. REQUIRED
registry-url: "https://registry.npmjs.org"
- run: pnpm install
- run: git config --global user.name machineuser
- run: git config --global user.email [email protected]
- run: |
PACKAGE_VERSION=$(node -p "require('./package.json').version")
BUMPED_VERSION=$(node -p "require('semver').inc('$PACKAGE_VERSION', '${{ github.event.inputs.newversion }}')")
# Update package.json with the new version
node -e "const fs = require('fs'); const package = JSON.parse(fs.readFileSync('./package.json')); package.version = '$BUMPED_VERSION'; fs.writeFileSync('./package.json', JSON.stringify(package, null, '\t') + '\n');"
git commit . -m "🔖 @huggingface/gearhash-wasm $BUMPED_VERSION"
git tag "gearhash-wasm-v$BUMPED_VERSION"
- run: pnpm publish --no-git-checks .
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- run: (git pull --rebase && git push --follow-tags) || (git pull --rebase && git push --follow-tags)
# hack - reuse actions/setup-node@v3 just to set a new registry
- uses: actions/setup-node@v3
with:
node-version: "20"
registry-url: "https://npm.pkg.github.com"
# Disable for now, until github supports PATs for writing github packages (https://github.com/github/roadmap/issues/558)
# - run: pnpm publish --no-git-checks .
# env:
# NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
64 changes: 64 additions & 0 deletions .github/workflows/splitmix64-wasm-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Splitmix64 WASM - Version and Release

on:
workflow_dispatch:
inputs:
newversion:
type: choice
description: "Semantic Version Bump Type"
default: patch
options:
- patch
- minor
- major

concurrency:
group: "push-to-main"

defaults:
run:
working-directory: packages/splitmix64-wasm

jobs:
version_and_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
# Needed to push the tag and the commit on the main branch, otherwise we get:
# > Run git push --follow-tags
# remote: error: GH006: Protected branch update failed for refs/heads/main.
# remote: error: Changes must be made through a pull request. Required status check "lint" is expected.
token: ${{ secrets.BOT_ACCESS_TOKEN }}
- run: npm install -g corepack@latest && corepack enable
- uses: actions/setup-node@v3
with:
node-version: "20"
cache: "pnpm"
cache-dependency-path: |
packages/splitmix64-wasm/pnpm-lock.yaml
# setting a registry enables the NODE_AUTH_TOKEN env variable where we can set an npm token. REQUIRED
registry-url: "https://registry.npmjs.org"
- run: pnpm install
- run: git config --global user.name machineuser
- run: git config --global user.email [email protected]
- run: |
PACKAGE_VERSION=$(node -p "require('./package.json').version")
BUMPED_VERSION=$(node -p "require('semver').inc('$PACKAGE_VERSION', '${{ github.event.inputs.newversion }}')")
# Update package.json with the new version
node -e "const fs = require('fs'); const package = JSON.parse(fs.readFileSync('./package.json')); package.version = '$BUMPED_VERSION'; fs.writeFileSync('./package.json', JSON.stringify(package, null, '\t') + '\n');"
git commit . -m "🔖 @huggingface/splitmix64-wasm $BUMPED_VERSION"
git tag "splitmix64-wasm-v$BUMPED_VERSION"
- run: pnpm publish --no-git-checks .
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- run: (git pull --rebase && git push --follow-tags) || (git pull --rebase && git push --follow-tags)
# hack - reuse actions/setup-node@v3 just to set a new registry
- uses: actions/setup-node@v3
with:
node-version: "20"
registry-url: "https://npm.pkg.github.com"
# Disable for now, until github supports PATs for writing github packages (https://github.com/github/roadmap/issues/558)
# - run: pnpm publish --no-git-checks .
# env:
# NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
64 changes: 64 additions & 0 deletions .github/workflows/xetchunk-wasm-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Xetchunk WASM - Version and Release

on:
workflow_dispatch:
inputs:
newversion:
type: choice
description: "Semantic Version Bump Type"
default: patch
options:
- patch
- minor
- major

concurrency:
group: "push-to-main"

defaults:
run:
working-directory: packages/xetchunk-wasm

jobs:
version_and_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
# Needed to push the tag and the commit on the main branch, otherwise we get:
# > Run git push --follow-tags
# remote: error: GH006: Protected branch update failed for refs/heads/main.
# remote: error: Changes must be made through a pull request. Required status check "lint" is expected.
token: ${{ secrets.BOT_ACCESS_TOKEN }}
- run: npm install -g corepack@latest && corepack enable
- uses: actions/setup-node@v3
with:
node-version: "20"
cache: "pnpm"
cache-dependency-path: |
packages/xetchunk-wasm/pnpm-lock.yaml
# setting a registry enables the NODE_AUTH_TOKEN env variable where we can set an npm token. REQUIRED
registry-url: "https://registry.npmjs.org"
- run: pnpm install
- run: git config --global user.name machineuser
- run: git config --global user.email [email protected]
- run: |
PACKAGE_VERSION=$(node -p "require('./package.json').version")
BUMPED_VERSION=$(node -p "require('semver').inc('$PACKAGE_VERSION', '${{ github.event.inputs.newversion }}')")
# Update package.json with the new version
node -e "const fs = require('fs'); const package = JSON.parse(fs.readFileSync('./package.json')); package.version = '$BUMPED_VERSION'; fs.writeFileSync('./package.json', JSON.stringify(package, null, '\t') + '\n');"
git commit . -m "🔖 @huggingface/xetchunk-wasm $BUMPED_VERSION"
git tag "xetchunk-wasm-v$BUMPED_VERSION"
- run: pnpm publish --no-git-checks .
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- run: (git pull --rebase && git push --follow-tags) || (git pull --rebase && git push --follow-tags)
# hack - reuse actions/setup-node@v3 just to set a new registry
- uses: actions/setup-node@v3
with:
node-version: "20"
registry-url: "https://npm.pkg.github.com"
# Disable for now, until github supports PATs for writing github packages (https://github.com/github/roadmap/issues/558)
# - run: pnpm publish --no-git-checks .
# env:
# NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Loading
Loading