Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 11 additions & 17 deletions skills/nemo-mbridge-mlm-bridge-training/BENCHMARK.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s
## Evaluation Summary

- Skill: `nemo-mbridge-mlm-bridge-training`
- Evaluation date: 2026-06-02
- Evaluation date: 2026-07-15
- NVSkills-Eval profile: `external`
- Environment: `local`
- Environment: `astra-sandbox`
- Dataset: 1 evaluation tasks
- Attempts per task: 2
- Attempts per task: 1
- Pass threshold: 50%
- Overall verdict: PASS

Expand Down Expand Up @@ -54,34 +54,28 @@ Task composition is derived from the evaluation dataset when possible. Entries w

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 2 | 100% (+0%) | 100% (+0%) |
| Correctness | 2 | 100% (+0%) | 88% (+0%) |
| Discoverability | 2 | 100% (+0%) | 62% (+0%) |
| Effectiveness | 2 | 100% (+0%) | 100% (+0%) |
| Efficiency | 2 | 93% (-0%) | 60% (-0%) |
| Security | 1 | 100% (+0%) | 100% (+0%) |
| Correctness | 1 | 100% (+100%) | 97% (+25%) |
| Discoverability | 1 | 100% (+100%) | 97% (+64%) |
| Effectiveness | 1 | 100% (+95%) | 100% (+0%) |
| Efficiency | 1 | 94% (+67%) | 96% (+59%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 12 total findings.
Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 4 total findings.

Top findings:

- MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (`skills/nemo-mbridge-mlm-bridge-training/SKILL.md`)
- MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/nemo-mbridge-mlm-bridge-training/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-mlm-bridge-training/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-mlm-bridge-training/SKILL.md`)
- MEDIUM SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-mlm-bridge-training/SKILL.md`)
- LOW SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-mlm-bridge-training/card.yaml`)

## Tier 2: Deduplication Summary

Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings.

Notable observations:

- Context Deduplication: Collected 1 file(s)
- Inter-Skill Deduplication: Parsed skill 'nemo-mbridge-mlm-bridge-training': 145 char description
This tier was not run or did not produce findings in this report.

## Publication Recommendation

Expand Down
6 changes: 3 additions & 3 deletions skills/nemo-mbridge-mlm-bridge-training/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ uv run python -m torch.distributed.run --nproc_per_node=1 \
--recipe vanilla_gpt_pretrain_config \
model.num_layers=2 model.hidden_size=256 \
model.num_attention_heads=4 model.ffn_hidden_size=1024 \
model.seq_length=512 dataset.sequence_length=512 \
model.seq_length=512 dataset.seq_length=512 \
train.train_iters=10 train.global_batch_size=32 train.micro_batch_size=4 \
validation.eval_interval=10 validation.eval_iters=2 \
optimizer.lr=3e-4 optimizer.min_lr=3e-5 \
Expand Down Expand Up @@ -99,7 +99,7 @@ uv run python -m torch.distributed.run --nproc_per_node=2 \
model.tensor_model_parallel_size=2 model.sequence_parallel=true \
model.num_layers=4 model.hidden_size=256 \
model.num_attention_heads=4 model.ffn_hidden_size=1024 \
model.seq_length=1024 dataset.sequence_length=1024 \
model.seq_length=1024 dataset.seq_length=1024 \
train.train_iters=10 train.global_batch_size=16 train.micro_batch_size=2 \
validation.eval_interval=10 validation.eval_iters=2 \
scheduler.lr_warmup_iters=2 scheduler.lr_decay_iters=10 \
Expand Down Expand Up @@ -169,7 +169,7 @@ git submodule update --init 3rdparty/Megatron-LM
value, also set `scheduler.lr_warmup_iters` and `scheduler.lr_decay_iters`
or you get an assertion error.

5. **Use `dataset.sequence_length`** in CLI overrides, not `dataset.seq_length`.
5. **Use `dataset.seq_length`** in CLI overrides for both pretraining and fine-tuning datasets.

6. **MoE OOM**: Large MoE models require full activation recomputation and
typically multi-node EP. TP does NOT reduce per-GPU expert memory.
Expand Down
2 changes: 1 addition & 1 deletion skills/nemo-mbridge-mlm-bridge-training/card.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ recommended_path:
known_constraints:
- MLM requires --eval-iters and --eval-interval (no defaults).
- Bridge scheduler asserts lr_warmup_iters < lr_decay_iters.
- Use dataset.sequence_length (not dataset.seq_length) in CLI overrides.
- Use dataset.seq_length in CLI overrides for both pretraining and fine-tuning datasets.
- MLM requires PYTHONPATH to include 3rdparty/Megatron-LM.
- Bridge auto-resumes from nemo_experiments/ if previous checkpoint exists.
known_limitations:
Expand Down
29 changes: 18 additions & 11 deletions skills/nemo-mbridge-mlm-bridge-training/skill-card.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
## Description: <br>
Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data, covering correlation testing, available recipes, and multi-GPU examples. <br>
Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data. Covers correlation testing, available recipes, and multi-GPU examples. <br>

This skill is ready for commercial/non-commercial use. <br>

Expand All @@ -9,22 +9,29 @@ NVIDIA <br>
### License/Terms of Use: <br>
Apache 2.0 <br>
## Use Case: <br>
Developers and engineers running Megatron-LM or Megatron Bridge training, comparing MLM vs Bridge loss curves, translating MLM CLI args to Bridge config, or debugging correlation divergences. <br>
Developers and engineers running Megatron-LM and Megatron Bridge training, comparing MLM vs Bridge loss curves, translating MLM CLI arguments to Bridge config, and investigating training correlation issues. <br>

### Deployment Geography for Use: <br>
Global <br>

## Requirements / Dependencies: <br>
**Requires API Key or External Credential:** [Not Specified] <br>
**Credential Type(s):** [None identified] <br>

Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>

## Known Risks and Mitigations: <br>
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br>
Mitigation: Review and scan skill before deployment. <br>

## Reference(s): <br>
- [Megatron-LM to Megatron Bridge Guide](docs/megatron-lm-to-megatron-bridge.md) <br>
- [Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/) <br>
- [Bridge Training Entry Point (run_recipe.py)](scripts/training/run_recipe.py) <br>
- [NeMo Megatron Bridge Documentation](https://docs.nvidia.com/nemo/megatron-bridge/latest/) <br>


## Skill Output: <br>
**Output Type(s):** [Shell commands, Configuration instructions, Analysis] <br>
**Output Type(s):** [Shell commands, Configuration instructions] <br>
**Output Format:** [Markdown with inline bash code blocks] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>
Expand All @@ -36,7 +43,7 @@ Mitigation: Review and scan skill before deployment. <br>


## Evaluation Tasks: <br>
Evaluated against 1 evaluation task (positive skill-activation) with 2 attempts per task via NVSkills-Eval external profile. <br>
Evaluated against 1 internal skill task (positive activation) in astra-sandbox environment using NVSkills-Eval external profile. <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
Expand All @@ -60,14 +67,14 @@ Underlying evaluation signals used in this run: <br>
## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 2 | 100% (+0%) | 100% (+0%) |
| Correctness | 2 | 100% (+0%) | 88% (+0%) |
| Discoverability | 2 | 100% (+0%) | 62% (+0%) |
| Effectiveness | 2 | 100% (+0%) | 100% (+0%) |
| Efficiency | 2 | 93% (-0%) | 60% (-0%) |
| Security | 1 | 100% (+0%) | 100% (+0%) |
| Correctness | 1 | 100% (+100%) | 97% (+25%) |
| Discoverability | 1 | 100% (+100%) | 97% (+64%) |
| Effectiveness | 1 | 100% (+95%) | 100% (+0%) |
| Efficiency | 1 | 94% (+67%) | 96% (+59%) |

## Skill Version(s): <br>
b0f64d72 (source: git SHA, committed 2026-06-02) <br>
1.0.0+b7643bd (source: pyproject.toml) <br>

## Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
Expand Down
2 changes: 1 addition & 1 deletion skills/nemo-mbridge-mlm-bridge-training/skill.oms.sig
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLW1sbS1icmlkZ2UtdHJhaW5pbmciLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiMjA3Njg0OGM0NTQ3YzU0MTg0YjI0MjE2ZTM1Y2NmODkxYWQ1MTYxYTEzZjVhNjU0YWU3MjQ2NmIyMTc4YWM1ZCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICJlMDFmNjVmZjk1MGM1ZTBhMTM0YzM1Yzg0ZmI0ODQ0YjkxOTBlNDhmMTMyZWNhMTVkYWZiZDViNzkxNTA5ZDQ1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjJjYWVlMzk1NDA5YWNhNTZiNWMzZGJkZTkwZDE0MWFjMzc4YmFlMWE4ZTQ0NGUyM2Q3M2U2MWExMWRhODc5ZGQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiY2FyZC55YW1sIiwKICAgICAgICAiZGlnZXN0IjogImI5MGRmYWE0MmQyMGUxNTJjYTI2YjBlMDNkOGIxZjY5YjU2YzM2Yjg0YmJmYjA1MDZlMmQ4Y2Y4MTJmOWYxNzciLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImRpZ2VzdCI6ICIyM2QyNmQzMWM0ZGQ0M2Y0NjQ2OTUyZDRiZjk5NDhlMTdhNjAwYWFkZTczN2MyYzM1N2YwZDdiYzA3ZDM5MDk4IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiMWY0YWJiMGUxNjZiODlhMzg2ZDBhMDY3NDU1M2M3OWNmYWQyMjUxNzA3ODI4OWNlNGNkZDM2MDMwMDZmOTZkOSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRpZ25vcmUiCiAgICAgIF0sCiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIsCiAgICAgICJtZXRob2QiOiAiZmlsZXMiCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGQCMG3Jjn1qc0DtljajZCYqos3Hxo6d/dh6moV6VqpaH6jldahsi0Li7SGbF6w26zt3SwIwP9xszKA0NfyAI9ibT1gzuVegSJ6Z8vTvxV3LxvDU9lpuHryNb3QPn28ikUP1hGRO","keyid":""}]}}
{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1tYnJpZGdlLW1sbS1icmlkZ2UtdHJhaW5pbmciLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiYzNhNDRjMWFhMmI5ODJmODE0ZDhjOTZkMjE4Y2VmYWNlNzBlODJjZjkyN2Q0NGJmYjFiZmEzZWVjZDc4MzIzMyIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiNjA1MzkxZjI2ZjAwOTA1NDk4NTMwOTUxMjkxODFhZTIwMTMyMjRjMjg0MTFmNmYxOGY1N2UyYzY1ODI2ZWQ2ZSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiN2VhZTk3NmViNDlhNzNlODdkMWU3N2E1Mjk1YzNhNzllNGYxZmFjN2E3MGRkMDY0M2NiZmZjNWQ0ZDM4MDFjMSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJlODMwNmU4NjQ0MGFkN2YzYjgwN2ZlNzQ1ODI5MjhmMzRjMmEzMDUwZDUxMGE3ZGMwZGRjNjk1OWVkOGU4MzYzIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiY2FyZC55YW1sIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICIyM2QyNmQzMWM0ZGQ0M2Y0NjQ2OTUyZDRiZjk5NDhlMTdhNjAwYWFkZTczN2MyYzM1N2YwZDdiYzA3ZDM5MDk4IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiOWU4MGMzYTdhZDIxNWQ4NzgyMzc5ZTZhZDE4NTMxMmNjYTJiNjQ4NmM2MGU5ZDlhODExYWU2NzQ0NThjY2E2OSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0aWdub3JlIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGh1YiIKICAgICAgXSwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJtZXRob2QiOiAiZmlsZXMiCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMHMrwkKx/FrVi3+7AV5le48dj3xnni58TPn7yeMDcTSLRvneLkdSUKRZy76viF0DdQIxAJ2aOHu5OJ+WVlETrTZJk8a/gu82+rZgW5WduCAn5TmYDL0IljX2jZUhDhbMKw7RBQ==","keyid":""}]}}
10 changes: 5 additions & 5 deletions skills/nemo-mbridge-perf-sequence-packing/BENCHMARK.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s
## Evaluation Summary

- Skill: `nemo-mbridge-perf-sequence-packing`
- Evaluation date: 2026-07-14
- Evaluation date: 2026-07-15
- NVSkills-Eval profile: `external`
- Environment: `astra-sandbox`
- Dataset: 1 evaluation tasks
Expand Down Expand Up @@ -55,10 +55,10 @@ Task composition is derived from the evaluation dataset when possible. Entries w
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 1 | 100% (+0%) | 100% (+0%) |
| Correctness | 1 | 100% (+100%) | 94% (+19%) |
| Discoverability | 1 | 100% (+100%) | 81% (+44%) |
| Effectiveness | 1 | 96% (+96%) | 89% (+7%) |
| Efficiency | 1 | 94% (+67%) | 70% (+27%) |
| Correctness | 1 | 100% (+100%) | 97% (+42%) |
| Discoverability | 1 | 100% (+100%) | 97% (+69%) |
| Effectiveness | 1 | 95% (+95%) | 94% (+48%) |
| Efficiency | 1 | 94% (+67%) | 96% (+77%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

Expand Down
Loading