Skip to content

feat: add pickle-free safetensors checkpoints - #26

Merged
opensource-SantanderAI merged 1 commit into
SantanderAI:mainfrom
rschumann:feat/safetensors-checkpoints
Jul 23, 2026
Merged

feat: add pickle-free safetensors checkpoints#26
opensource-SantanderAI merged 1 commit into
SantanderAI:mainfrom
rschumann:feat/safetensors-checkpoints

Conversation

@rschumann

Copy link
Copy Markdown
Contributor

Summary

  • replace new PyTorch checkpoint writes with a single pickle-free safetensors file containing versioned JSON metadata
  • keep .pt and .pth checkpoints read-only through torch.load(..., weights_only=True) and add LinearAdapter.migrate_checkpoint()
  • validate metadata, configuration, tensor keys, and tensor shapes strictly, with no fallback from malformed safetensors to the legacy loader
  • fail fast on configured legacy output paths, update CLI defaults and examples, and document the breaking write-format change

Type of change

  • feat - new feature
  • fix - bug fix
  • docs - documentation only
  • test - adding or updating tests
  • refactor - code refactoring (no feature/fix)
  • ci - CI/CD changes
  • chore - maintenance

Linked issues

Closes #20

Checklist

  • Commit messages follow Conventional Commits
  • Tests added or updated (pytest tests/ -v --cov=linear_adapter_trainer)
  • Coverage stays at or above 80%
  • Lint/format/type pass locally (ruff check . && black --check . && mypy linear_adapter_trainer)
  • Documentation updated (README, CHANGELOG, docstrings) because behavior changed
  • The test suite still runs fully offline with no network or API keys
  • I have signed the CLA (not yet requested; sign through CLA Assistant if prompted on the first pull request, subject to separate approval)
  • I agree to follow the project's Code of Conduct

Notes for reviewers

New writes now require the .safetensors extension. Existing .pt and .pth
checkpoints remain readable through the restricted legacy path and can be
migrated explicitly. Parser selection is extension-based, so a corrupt or
renamed safetensors artifact is never passed to torch.load.

The implementation uses safetensors' native string metadata for a single-file
artifact. The metadata contains format version 1 and canonical JSON for the
five AdapterConfig fields. Unknown, missing, mistyped, or invalid fields are
rejected before tensor loading. Writes use a temporary file in the destination
directory, fsync, and os.replace.

Atomic writes use tempfile.mkstemp, so new checkpoints are owner-only
(0600) on POSIX. The Windows durability path was reviewed but not executed
because upstream CI is Linux-only. The updated notebook cell's output was
cleared rather than regenerated without live API access.

Linux verification used Python 3.12.13, CPU-only Torch 2.13.0, and safetensors
0.8.0. Results: 97 tests passed, total coverage is 80%, the checkpoint module
has 100% coverage, and Ruff, Black, Mypy, build, pip-audit, dependency-license,
SPDX, internal-pattern, changed-code Bandit, secret, and private-pattern checks
passed.

@rschumann
rschumann requested a review from a team as a code owner July 23, 2026 11:06
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@rschumann

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Jul 23, 2026
@opensource-SantanderAI
opensource-SantanderAI merged commit 29c8b22 into SantanderAI:main Jul 23, 2026
9 of 10 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 23, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harden checkpoint format: remove pickle from adapter serialization (safetensors)

2 participants