Code

Amgen's AMPLIFY Port#442

Code

Draft

ynashed wants to merge 116 commits intoNVIDIA:mainNVIDIA/bionemo-framework:mainfrom ynashed:ynashed/v2-main/amplifyynashed/bionemo-framework:ynashed/v2-main/amplifyCopy head branch name to clipboard

+3,210

Commits

Commits on Sep 11, 2024

• adding bionemo-scclip to sub-packages

  ynashed

  committed
  9755977

  Copy full SHA for 9755977

Commits on Sep 30, 2024

• Initial model change

  ynashed

  committed
  843ce40

  Copy full SHA for 843ce40

Commits on Oct 3, 2024

• Add bionemo-amplify sub-package and its requirements

  ynashed

  committed
  9fe70d6

  Copy full SHA for 9fe70d6

• Added AMPLIFY tokenizer

  ynashed

  committed
  e21c692

  Copy full SHA for e21c692

Commits on Oct 4, 2024

• Added AMPLIFY dataset and dataloader

  ynashed

  committed
  2e9e299

  Copy full SHA for 2e9e299

• Finalizing amplify model and train script

  ynashed

  committed
  8b41f03

  Copy full SHA for 8b41f03

• Add bionemo-esm2 to local requirements

  ynashed

  committed
  82928a4

  Copy full SHA for 82928a4

• Update biobert_spec_option to use esm2_bert_layer_local_spec

  ynashed

  committed
  2d0181e

  Copy full SHA for 2d0181e

• Fix syntax error

  ynashed

  committed
  87b259f

  Copy full SHA for 87b259f

• Update lr_scheduler import in amplify_pretrain.py

  ynashed

  committed
  fa8daad

  Copy full SHA for fa8daad

• Update amplify_pretrain.py to use hf_dataset_name instead of individual file paths

  ynashed

  committed
  794a40b

  Copy full SHA for 794a40b

Commits on Oct 5, 2024

• Fixing errors related to hf_dataset_name

  ynashed

  committed
  c1a03d2

  Copy full SHA for c1a03d2

• Update optimizer and lr_scheduler in amplify_pretrain.py

  ynashed

  committed
  94b9649

  Copy full SHA for 94b9649

• Bugfixes

  ynashed

  committed
  d8e4d1a

  Copy full SHA for d8e4d1a

Commits on Oct 8, 2024

• Merge pull request #1 from NVIDIA/main

  Show description for 552a952
  ynashed

  authored
  552a952

  Copy full SHA for 552a952

• adding bionemo-scclip to sub-packages

  ynashed

  committed
  0cbc475

  Copy full SHA for 0cbc475

• Initial model change

  ynashed

  committed
  a1e323f

  Copy full SHA for a1e323f

• Add bionemo-amplify sub-package and its requirements

  ynashed

  committed
  bd7041c

  Copy full SHA for bd7041c

• Added AMPLIFY tokenizer

  ynashed

  committed
  897601e

  Copy full SHA for 897601e

• Added AMPLIFY dataset and dataloader

  ynashed

  committed
  1dc780c

  Copy full SHA for 1dc780c

• Finalizing amplify model and train script

  ynashed

  committed
  88ebe21

  Copy full SHA for 88ebe21

• Add bionemo-esm2 to local requirements

  ynashed

  committed
  ee6e91b

  Copy full SHA for ee6e91b

• Update biobert_spec_option to use esm2_bert_layer_local_spec

  ynashed

  committed
  a7f8ff4

  Copy full SHA for a7f8ff4

• Fix syntax error

  ynashed

  committed
  1a1c4e8

  Copy full SHA for 1a1c4e8

• Update lr_scheduler import in amplify_pretrain.py

  ynashed

  committed
  3c9ffc7

  Copy full SHA for 3c9ffc7

• Update amplify_pretrain.py to use hf_dataset_name instead of individual file paths

  ynashed

  committed
  52809c7

  Copy full SHA for 52809c7

• Fixing errors related to hf_dataset_name

  ynashed

  committed
  9a4aec8

  Copy full SHA for 9a4aec8

• Update optimizer and lr_scheduler in amplify_pretrain.py

  ynashed

  committed
  d6eada9

  Copy full SHA for d6eada9

• Bugfixes

  ynashed

  committed
  0901c5d

  Copy full SHA for 0901c5d

• Update tach.toml to include bionemo.amplify module and its dependencies

  ynashed

  committed
  d261500

  Copy full SHA for d261500

Commits on Oct 10, 2024

• Updated bionemo-amplify to sync with upstream esm2 changes

  ynashed

  committed
  579f16a

  Copy full SHA for 579f16a

• Ignore test_experiment directory in git

  ynashed

  committed
  0c9de08

  Copy full SHA for 0c9de08

• solving merge conflicts

  ynashed

  committed
  abfa9da

  Copy full SHA for abfa9da

• Update BioNeMoAMPLIFYTokenizer to use EsmTokenizer

  ynashed

  committed
  3820690

  Copy full SHA for 3820690

• Update BioNeMoAMPLIFYTokenizer to use chandar-lab/AMPLIFY_350M

  ynashed

  committed
  059cc8c

  Copy full SHA for 059cc8c

• Update BioNeMoAMPLIFYTokenizer to fix serialization issue

  ynashed

  committed
  b8fd917

  Copy full SHA for b8fd917

• Fix range for random_tokens in AMPLIFYMaskedResidueDataset

  ynashed

  committed
  80947b0

  Copy full SHA for 80947b0

• Refactor index variable in AMPLIFYMaskedResidueDataset's __getitem__ method

  ynashed

  committed
  f8eac7c

  Copy full SHA for f8eac7c

Commits on Oct 14, 2024

• Adding AMPLIFY specific config parameters

  ynashed

  committed
  121ff5f

  Copy full SHA for 121ff5f

Commits on Oct 17, 2024

• Amplify doesn't inherit from ESM2Model anymore

  ynashed

  committed
  c758656

  Copy full SHA for c758656

Commits on Oct 19, 2024

• removed extra layernorm, added gradient clipping, configure cosine lr annealing

  ynashed

  committed
  1e36af1

  Copy full SHA for 1e36af1

Commits on Oct 22, 2024

• reducing attention block ffn_hidden_size to match the paper

  ynashed

  committed
  273cf6a

  Copy full SHA for 273cf6a

• Dataset resampling with MultiEpochDatasetResampler

  ynashed

  committed
  26b12a0

  Copy full SHA for 26b12a0

• Merge branch 'NVIDIA:main' into v2-main

  ynashed

  authored
  21083c0

  Copy full SHA for 21083c0

• cast np.int64 to int in dataset __getitem__

  ynashed

  committed
  e47def5

  Copy full SHA for e47def5

• Merge branch 'v2-main' into ynashed/v2-main/amplify

  ynashed

  committed
  522e028

  Copy full SHA for 522e028

• Update amplify to match latest esm2 code

  ynashed

  committed
  eb2d658

  Copy full SHA for eb2d658

• Revert to PRNGResampleDataset

  ynashed

  committed
  b822461

  Copy full SHA for b822461

Commits on Oct 23, 2024

• optimize multi_epoch_dataset for constant memory and space usage

  pstjohn

  committed
  7f40015

  Copy full SHA for 7f40015

• Merge pull request #2 from NVIDIA/pstjohn/main/optimize-multi-epoch-dataset

  Show description for e09db4e
  ynashed

  authored
  e09db4e

  Copy full SHA for e09db4e

• Trying out upstream MultiEpochDatasetResampler optimization

  ynashed

  committed
  70284e0

  Copy full SHA for 70284e0

• Added dataset_subset to AMPLIFYMaskedResidueDataset

  ynashed

  committed
  89166e4

  Copy full SHA for 89166e4

Commits on Nov 2, 2024

• Changed defaults to 120M Model. Added final_step to lr scheduler (cosine decay)

  ynashed

  committed
  f699da2

  Copy full SHA for f699da2

Commits on Nov 4, 2024

• enabled bf16 in the optimizer

  ynashed

  committed
  97b4bf6

  Copy full SHA for 97b4bf6

• LR scheduler warmup starts from min_lr 0 by default

  ynashed

  committed
  77040d5

  Copy full SHA for 77040d5

• Fixing cosine lr scheduler to match HF implementation

  ynashed

  committed
  af44848

  Copy full SHA for af44848

Commits on Nov 5, 2024

• Turning off CosineAnnealingScheduler constan_steps

  ynashed

  committed
  c52d5ba

  Copy full SHA for c52d5ba

Commits on Nov 6, 2024

• RandomMaskStrategy defaults to AMINO_ACIDS_ONLY

  ynashed

  committed
  2305b17

  Copy full SHA for 2305b17

• Make sure <mask> token exists in masked sequence

  ynashed

  committed
  429bbdc

  Copy full SHA for 429bbdc

• Forgot self. (doh)

  ynashed

  committed
  c5ef9d4

  Copy full SHA for c5ef9d4

Commits on Nov 7, 2024

• revert the masking check

  ynashed

  committed
  c65f86e

  Copy full SHA for c65f86e

• Turning off weight decay

  ynashed

  committed
  2f1b149

  Copy full SHA for 2f1b149

Commits on Nov 8, 2024

• [WIP] fix tests

  ynashed

  committed
  16d0a30

  Copy full SHA for 16d0a30

• Merge remote-tracking branch 'upstream/main'

  ynashed

  committed
  3260d37

  Copy full SHA for 3260d37

• Merge branch 'main' into ynashed/v2-main/amplify

  ynashed

  committed
  ece8de4

  Copy full SHA for ece8de4

• esm2 updates

  ynashed

  committed
  d64740e

  Copy full SHA for d64740e

• Fixes after testing

  ynashed

  committed
  82de2ab

  Copy full SHA for 82de2ab

• adding log-every-n-steps argument

  ynashed

  committed
  cdba113

  Copy full SHA for cdba113

Commits on Nov 12, 2024

• Merge remote-tracking branch 'upstream/main'

  ynashed

  committed
  6c52266

  Copy full SHA for 6c52266

• Merge branch 'main' into ynashed/v2-main/amplify

  ynashed

  committed
  9b7d5ff

  Copy full SHA for 9b7d5ff

Commits on Nov 13, 2024

• added nsys profiling arguments

  ynashed

  committed
  efc53eb

  Copy full SHA for efc53eb

• removed slowdown in dataset getitem

  ynashed

  committed
  e803606

  Copy full SHA for e803606

Commits on Nov 14, 2024

• Trying differemt optimizer and model configs

  ynashed

  committed
  3f649ab

  Copy full SHA for 3f649ab

• roll back config changes

  ynashed

  committed
  40566ec

  Copy full SHA for 40566ec

Commits on Nov 15, 2024

• Removed abandoned bionemo-scclip

  ynashed

  committed
  2a2cbac

  Copy full SHA for 2a2cbac

Commits on Dec 10, 2024

• Patching Megatron-LM to include pytorch optimizers as default

  ynashed

  committed
  2f051f8

  Copy full SHA for 2f051f8

• Adding run:ai submit scriots [WIP]

  ynashed

  committed
  d10ab06

  Copy full SHA for d10ab06

• Added Mehatron optimizer patch file

  ynashed

  committed
  3e7f959

  Copy full SHA for 3e7f959

Commits on Dec 12, 2024

• changes to match NVIDIA/NeMo#11252

  ynashed

  committed
  50025eb

  Copy full SHA for 50025eb

• Adding WANDB_API_KEY run:ai secret

  ynashed

  committed
  4280418

  Copy full SHA for 4280418

• Switch back to pytorch_lightning.callbacks import

  ynashed

  committed
  5d03f8d

  Copy full SHA for 5d03f8d

Commits on Dec 13, 2024

• trying to get rid if loss spikes

  ynashed

  committed
  3339571

  Copy full SHA for 3339571

• pip installing megatron-lm just for good measure

  ynashed

  committed
  faed442

  Copy full SHA for faed442

• faster weight decay

  ynashed

  committed
  85e1d78

  Copy full SHA for 85e1d78

• Changing core attention to default

  ynashed

  committed
  f98f02d

  Copy full SHA for f98f02d

• Trying training in fp32

  ynashed

  committed
  c161652

  Copy full SHA for c161652

• OOM, trying fp32-mixed

  ynashed

  committed
  192e121

  Copy full SHA for 192e121

• OOM, fp16-mixed

  ynashed

  committed
  5d37fea

  Copy full SHA for 5d37fea

Commits on Dec 14, 2024

• lower initial loss_scaling

  ynashed

  committed
  32aa173

  Copy full SHA for 32aa173

• fp16-mixed precision with constant loss scaling

  ynashed

  committed
  5927369

  Copy full SHA for 5927369

• loss_scale passed to the right class

  ynashed

  committed
  ce2298c

  Copy full SHA for ce2298c

• MegatronMixedPrecision class argument name fix

  ynashed

  committed
  77d3239

  Copy full SHA for 77d3239

• reverting back to bf16-mixed

  ynashed

  committed
  432acf0

  Copy full SHA for 432acf0

• Increasing adam_eps to try counter grad norm explosion

  ynashed

  committed
  572511b

  Copy full SHA for 572511b

• reverting adam_eps to 1e-8

  ynashed

  committed
  7dcd4e1

  Copy full SHA for 7dcd4e1

• Turning off fusions, turning on attention_softmax_in_fp32

  ynashed

  committed
  d7e1cc0

  Copy full SHA for d7e1cc0

Commits on Dec 16, 2024

• Use esm2 LM_Head and layernorm instead of RMSNorm

  ynashed

  committed
  a02b4ee

  Copy full SHA for a02b4ee

• Trying esm2_bert_layer_with_transformer_engine_spec

  ynashed

  committed
  98a62d7

  Copy full SHA for 98a62d7

• Allowing AmplifyConfig to accept esm2 bert spec

  ynashed

  committed
  a027e2b

  Copy full SHA for a027e2b

• Removing LM_head, using esm2_bert_layer_with_transformer_engine_spec, and keeping RMSNorm

  ynashed

  committed
  e7db7d3

  Copy full SHA for e7db7d3

Commits on Dec 23, 2024

• kubectl script changes to add random-mask-strategy argument

  ynashed

  committed
  0f7ef3f

  Copy full SHA for 0f7ef3f

• Merge remote-tracking branch 'upstream/main' into ynashed/v2-main/amplify

  ynashed

  committed
  d60e76d

  Copy full SHA for d60e76d

Commits on Dec 24, 2024

• Merge branch 'ynashed/amplify/runai' into ynashed/v2-main/amplify

  ynashed

  committed
  62caa27

  Copy full SHA for 62caa27

Commits on Dec 25, 2024

• Updates after sync with upstream

  ynashed

  committed
  b1836eb

  Copy full SHA for b1836eb

Commits on Dec 30, 2024

• updating the pytorch_lightning import

  ynashed

  committed
  169e5bb

  Copy full SHA for 169e5bb

• Merge remote-tracking branch 'upstream/main'

  ynashed

  committed
  ec10b36

  Copy full SHA for ec10b36

• Merge branch 'main' into ynashed/v2-main/amplify

  ynashed

  committed
  b7e35ac

  Copy full SHA for b7e35ac

• Merge remote-tracking branch 'upstream/main' into ynashed/v2-main/amplify

  ynashed

  committed
  d0b837d

  Copy full SHA for d0b837d

• bump NeMo version to match upstream

  ynashed

  committed
  f6e0f37

  Copy full SHA for f6e0f37

Commits on Dec 31, 2024

• Add handling for special token IDs in BioNeMoAMPLIFYTokenizer

  ynashed

  committed
  8f15b80

  Copy full SHA for 8f15b80

• Add DDP configuration options for gradient reduction and parameter gathering

  ynashed

  committed
  0107bc6

  Copy full SHA for 0107bc6

• Adding train script entrypoint

  ynashed

  committed
  1a3ffa9

  Copy full SHA for 1a3ffa9

• Conforming with upstream changes

  ynashed

  committed
  7e8cddf

  Copy full SHA for 7e8cddf

• Removing runai scripts from git

  ynashed

  committed
  5942afa

  Copy full SHA for 5942afa

Commits on Jan 1, 2025

• AMPLIFYConfig::__post_init__ was called twice leading to decreased model size. Moved rescaling ffn_hidden_size to the AMPLIFYModel::__init_

  ynashed

  committed
  cbf3535

  Copy full SHA for cbf3535

• Disable distributed optimizer in training script

  ynashed

  committed
  7fe9c46

  Copy full SHA for 7fe9c46