Skip to content

Debug training runs example #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 111 commits into
base: main
Choose a base branch
from
Draft

Debug training runs example #11

wants to merge 111 commits into from

Conversation

LeoRoccoBreedt
Copy link
Contributor

@LeoRoccoBreedt LeoRoccoBreedt commented Apr 16, 2025 •
edited
Loading

Description

Include a summary of the changes and the related issue.

Related to: <ClickUp/JIRA task name>

Any expected test failures?

Add a [X] to relevant checklist items

❔ This change

  • adds a new feature
  • fixes breaking code
  • is cosmetic (refactoring/reformatting)

✔️ Pre-merge checklist

  • Refactored code (sourcery)
  • Tested code locally
  • Precommit installed and run before pushing changes
  • Added code to GitHub tests (notebooks, scripts)
  • Updated GitHub README
  • Updated the projects overview page on Notion

🧪 Test Configuration

  • OS: Windows
  • Python version: 3.11
  • Neptune version: neptune-scale 0.11.3
  • Affected libraries with version: torch, datasets

@LeoRoccoBreedt
feat: Added initial Pytorch example to monitor batching and per layer…
b837338 
… metrics
@LeoRoccoBreedt
refactor: Update introduction section for more clarity on notebook use
9994528
@LeoRoccoBreedt
chore: change how the custom run id gets automatically generated
223fde1
@LeoRoccoBreedt
chore: update instructions on how users can get and set their API tok…
ceb3a24 
…en and project name
@LeoRoccoBreedt
chore: update the introduction to be more foundation model training o…
39f5bb9 
…rientated
@LeoRoccoBreedt
chore: update dataset section with a better description
69fdfba
@LeoRoccoBreedt
refactor: update training loop where grads, norms and activations are…
1cde287 
… tracked
@LeoRoccoBreedt
refactor: update batch size and edit gradient norm logging code
2da28f4
@LeoRoccoBreedt
chore: add data file to ignore for pytorch example
123fea2
@LeoRoccoBreedt
refactor: update model architecture layers and update training loop
86b6d5d
@LeoRoccoBreedt
refactor: update accuracy calculation to not output the percentage
a39077a
@LeoRoccoBreedt
refactor: update model architecture layers, accuracy calculation and …
3e2d453 
…calculate gradient norms for batch (step) rather than epoch
@LeoRoccoBreedt
feat: Added a pytorch text-based example that is used to demonstrate …
8a3da57 
…debugging when building LLM's
@LeoRoccoBreedt
refactor: add validation and test loss calcualtion for each epoch
d03fd43
@LeoRoccoBreedt
refactor: update configs and parameters
b7f8e08
@LeoRoccoBreedt
refactor: update logged configs
b113a36
@LeoRoccoBreedt
refactor: calculate activations per layer
7258993
@LeoRoccoBreedt
refactor: add tracking for grad norms
ae330e6
@LeoRoccoBreedt
refactor: add gradient tracking per epoch
d4d160c
@LeoRoccoBreedt
chore: remove uneeded section
890bf26
@LeoRoccoBreedt
refactor: add fully connected layer to model for more complexity
ebd34c7
@LeoRoccoBreedt
chore: fix activation saving for layers
ae9abc6
@LeoRoccoBreedt
refactor: update packages for example
6f23761
@LeoRoccoBreedt
refactor: update dataset to be used in example
2c82a0f
@LeoRoccoBreedt
refactor: update evalution function that calculates the validation lo…
48451d6 
…sses using the data loader
@LeoRoccoBreedt
refactor: update training loop to work with new data
5b5fbdd
@LeoRoccoBreedt
chore: remove unused sections
8ff6f30
@LeoRoccoBreedt
chore: re-organize notebook layout
b32b27b
@LeoRoccoBreedt
chore: cleanup and add parameters in right place
4d659a2
@LeoRoccoBreedt
refactor: add all debugging metrics to the same dictionary variable
5e7ec6e
@LeoRoccoBreedt
chore: cleanup comments and add TODO for future improvements
75265ec
@LeoRoccoBreedt
refactor: update TorchWatcher to use a common _track_metrics() method
ebde389
@LeoRoccoBreedt
refactor: update the watch() method to accept boolen inputs for the m…
9f83fbf 
…etrics to track rather than named values
@LeoRoccoBreedt
feat: add ability to specify base_namespace and namespace during trai…
603af72 
…ning loop.

- more control on namespace logged during training
@LeoRoccoBreedt
chore: updare readme and example for namespace feature
6bb4879
@LeoRoccoBreedt
chore: pre-commit hooks changes
50b0a6d
@LeoRoccoBreedt
refactor: update example to use the improved TorchWatcher package
e67e1f6
@LeoRoccoBreedt
chore: pre-commit cleanup
a2d0d50
@LeoRoccoBreedt
feat: notebook tutorial on debugging training runs with Neptune
fcf3657
@LeoRoccoBreedt
chore: pre-commit changes
82f4d8b
@LeoRoccoBreedt
chore: update colab link for branch
8c56296
@LeoRoccoBreedt
refactor: keep notebook more self-contained for use in Colab
247679a
@LeoRoccoBreedt
fix: f string compatibility in Colab
408ba02
@LeoRoccoBreedt
fix: ensure model object is on correct device
314e8a2
@LeoRoccoBreedt
chore: Add header links to GH, Neptune and docs
62227bc 
These need to be updated the the final branch when merged
@LeoRoccoBreedt
Merge commit 'fc4bc5ee2d6e3297c8611991e60f372ea785d213' into lb/debug…
f64d54d 
…ging_model_training
@LeoRoccoBreedt
style: minor updates to markdown
505f41c
@LeoRoccoBreedt
chore: remove unused code
bed4899
@LeoRoccoBreedt
style: remove image reference
75885bb
@LeoRoccoBreedt
style: update ending steps and add button links to examples
33d3305
@LeoRoccoBreedt
style: update text for better readability
494379f
@LeoRoccoBreedt
style: Update markdown section of explanations for better readability…
0b8847b 
… and flow
@LeoRoccoBreedt
chore: pre-commit cleanup
69bdb16
@LeoRoccoBreedt
chore: removed unused code for this example
0fed412
@LeoRoccoBreedt
chore: update test yaml
ba14f8f
@LeoRoccoBreedt LeoRoccoBreedt self-assigned this Apr 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@LeoRoccoBreedt LeoRoccoBreedt

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@LeoRoccoBreedt @szaganek