Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
83cb844
remove component popping functionality
mivanit Oct 13, 2025
058055e
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 13, 2025
7856ea5
rename s1 to dataset in prep for refactor
mivanit Oct 13, 2025
94f82f3
wip
mivanit Oct 13, 2025
89bb997
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 20, 2025
0ccc65d
make format
mivanit Oct 20, 2025
b1e3729
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 20, 2025
064632c
format
mivanit Oct 20, 2025
b580455
typing fixes
mivanit Oct 20, 2025
3a206ed
allow specifying either config path or mrc cfg in pipeline cfg
mivanit Oct 21, 2025
eb831c0
[wip] reorg configs
mivanit Oct 21, 2025
89e5c36
added default `None` for slurm partition and job name prefix
mivanit Oct 21, 2025
8910bb4
refactor configs, add config tests
mivanit Oct 21, 2025
0b957f5
fix tests
mivanit Oct 21, 2025
7de545b
allow `None` or `-1` idx_in_ensemble
mivanit Oct 21, 2025
3d45ac4
whoops, wrong name on fixture
mivanit Oct 21, 2025
4adde10
fix idx passed in tests when not needed
mivanit Oct 21, 2025
189b64a
rename "mrc" -> "crc" in paths
mivanit Oct 21, 2025
57f445a
rename merge_run_config.py -> clustering_run_config.py
mivanit Oct 21, 2025
91f5348
fix pyright
mivanit Oct 21, 2025
11e5501
fix idx_in_ensemble being passed in tests
mivanit Oct 21, 2025
1d96054
rename cache dir 'merge_run_configs' -> 'clustering_run_configs'
mivanit Oct 21, 2025
45b2adc
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 21, 2025
2d65036
Merge branch 'clustering/add-mrc-inline-in-pipeline-cfg' into cluster…
mivanit Oct 21, 2025
9cbb52f
fix import
mivanit Oct 21, 2025
a1f1146
remove component popping
mivanit Oct 21, 2025
1e3fbb2
dont pass batch size, change not brought in here
mivanit Oct 21, 2025
f56db7f
Merge branch 'clustering/add-mrc-inline-in-pipeline-cfg' into cluster…
mivanit Oct 21, 2025
e86adc9
fix history_path extension and storage usage
mivanit Oct 22, 2025
b8bbb08
dev pipeline
mivanit Oct 22, 2025
733d47f
better config validation tests
mivanit Oct 22, 2025
2fa1f21
set default base output dir
mivanit Oct 22, 2025
eec80fb
wandb use run id for clustering, TODO for spd decomp
mivanit Oct 22, 2025
6098536
basedpyright 1.32.0 causes issues, esp w/ wandb
mivanit Oct 22, 2025
bd60f2c
Merge branch 'clustering/main' into clustering/add-mrc-inline-in-pipe…
mivanit Oct 23, 2025
f26e0fe
Merge branch 'clustering/add-mrc-inline-in-pipeline-cfg' into cluster…
mivanit Oct 23, 2025
40df505
remove idx_in_ensemble, always auto-assigned now
mivanit Oct 24, 2025
bd8a442
Merge branch 'clustering/main' into clustering/add-mrc-inline-in-pipe…
mivanit Oct 24, 2025
cf64a79
only allow passing clustering run config path, not inline
mivanit Oct 24, 2025
2a9f731
rename run_clustering_config_path -> clustering_run_config_path
mivanit Oct 24, 2025
e0a8551
Merge branch 'clustering/add-mrc-inline-in-pipeline-cfg' into cluster…
mivanit Oct 24, 2025
6dc14d8
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 24, 2025
7f676eb
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 24, 2025
6609223
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 27, 2025
04ae869
4 claudes in parallel
mivanit Oct 27, 2025
a18a807
make format
mivanit Oct 27, 2025
483e91c
type checking and compat fixes
mivanit Oct 27, 2025
6f6696a
old todos, removing in next commit
mivanit Oct 27, 2025
d7f0f68
remove old todos
mivanit Oct 27, 2025
a5505df
change recompute_costs_every behavior (None for single-batch)
mivanit Oct 27, 2025
3f07819
simplify & dedupe valid mask computation
mivanit Oct 27, 2025
76f7fcd
move non-nan count validity check
mivanit Oct 27, 2025
7c17247
wip
mivanit Oct 27, 2025
a7315c1
add back some type hints
mivanit Oct 27, 2025
2a18389
make format
mivanit Oct 27, 2025
28271c8
type hint fixes
mivanit Oct 27, 2025
62a7dd0
remove old wip file
mivanit Oct 27, 2025
24d4b8c
wip
mivanit Oct 27, 2025
8491c32
update STYLE.md
mivanit Oct 27, 2025
40df682
more updates to STYLE.md
mivanit Oct 27, 2025
8994490
refactor getting batches
mivanit Oct 27, 2025
c9a8007
wip
mivanit Oct 27, 2025
299aa48
pyright fix
mivanit Oct 27, 2025
f028eb4
wip
mivanit Oct 27, 2025
9f2e83f
wip, broken
mivanit Oct 27, 2025
ea0ff13
wip
mivanit Oct 29, 2025
5efdc2a
Merge branch 'clustering/main' into clustering/refactor-multi-batch
mivanit Oct 29, 2025
6376ffd
[temp] remove CI timeout
mivanit Oct 29, 2025
b4e8d82
change component indices dtype int16 -> int32
mivanit Oct 29, 2025
595c737
fix: allow dataset streaming in precompute_batches_for_single_run
mivanit Oct 29, 2025
791d0d1
"steaming" -> "streaming" typo fix
mivanit Oct 29, 2025
58ed42a
various dataset streaming fixes
mivanit Oct 29, 2025
b67c3ad
refactor to return dataloaders instead of single batches like before
mivanit Oct 29, 2025
1e2fe43
memory issues?
mivanit Oct 29, 2025
c886bd3
:(
mivanit Oct 29, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/checks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
runs-on: ubuntu-latest
container:
image: ghcr.io/${{ github.repository }}/ci-mpi:latest
timeout-minutes: 15
# timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v4
Expand Down
7 changes: 5 additions & 2 deletions STYLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,13 @@ If there's an assumption you're making while writing code, assert it.
- If you were wrong, then the code **should** fail.

## Type Annotations
- Use jaxtyping for tensor shapes (though for now we don't do runtime checking)
- **Always** use jaxtyping for tensor or numpy array shapes (though for now we don't do runtime checking)
- Always use the PEP 604 typing format of `|` for unions and `type | None` over `Optional`.
- Use `dict`, `list` and `tuple` not `Dict`, `List` and `Tuple`
- Don't add type annotations when they're redundant. (i.e. `my_thing: Thing = Thing()` or `name: str = "John Doe"`)
- Don't add type annotations only when they're redundant.
- i.e. `my_thing: Thing = Thing()` or `name: str = "John Doe"` don't need type annotations.
- however, `var = foo()` or `result = thing.bar()` should be annotated!
- FOR CLAUDE: don't worry about cleaning up unused imports, we do this automatically with ruff using `make format`

## Tensor Operations
- Try to use einops by default for clarity.
Expand Down
73 changes: 0 additions & 73 deletions TODO.md

This file was deleted.

Loading
Loading